Jul 06 20:56:49.808638 ip-10-0-136-207 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 06 20:56:49.808688 ip-10-0-136-207 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 06 20:56:49.808699 ip-10-0-136-207 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 06 20:56:49.808964 ip-10-0-136-207 systemd[1]: Failed to start Kubernetes Kubelet. Jul 06 20:56:59.826635 ip-10-0-136-207 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 06 20:56:59.826667 ip-10-0-136-207 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 7ea7ee8877134db0a97dec404bf3dba4 -- Jul 06 20:58:35.559359 ip-10-0-136-207 systemd[1]: Starting Kubernetes Kubelet... Jul 06 20:58:36.066248 ip-10-0-136-207 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 20:58:36.066248 ip-10-0-136-207 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 20:58:36.066248 ip-10-0-136-207 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 20:58:36.066248 ip-10-0-136-207 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 20:58:36.066248 ip-10-0-136-207 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 20:58:36.066248 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.947120 2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 20:58:36.066248 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954031 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:58:36.066248 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954051 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:58:36.066248 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954057 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:58:36.066248 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954062 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:58:36.066248 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954067 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:58:36.066248 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954070 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:58:36.066248 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954073 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:58:36.066248 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954076 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:58:36.032861 ip-10-0-136-207 systemd[1]: Started Kubernetes Kubelet. Jul 06 20:58:36.068860 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954079 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:58:36.068860 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954082 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:58:36.068860 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954085 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:58:36.068860 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954088 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:58:36.068860 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954091 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:58:36.068860 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954093 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:58:36.068860 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954096 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:58:36.068860 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954098 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:58:36.068860 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954101 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:58:36.068860 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954104 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:58:36.068860 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954108 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:58:36.068860 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954110 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:58:36.068860 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954113 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:58:36.068860 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954116 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:58:36.068860 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954119 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:58:36.068860 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954121 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:58:36.068860 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954123 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:58:36.068860 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954127 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:58:36.068860 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954130 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:58:36.068860 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954133 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:58:36.071119 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954136 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:58:36.071119 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954139 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:58:36.071119 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954142 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:58:36.071119 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954145 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:58:36.071119 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954148 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:58:36.071119 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954150 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:58:36.071119 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954153 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:58:36.071119 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954157 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:58:36.071119 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954160 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:58:36.071119 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954163 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:58:36.071119 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954166 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:58:36.071119 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954169 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:58:36.071119 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954171 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:58:36.071119 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954174 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:58:36.071119 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954176 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:58:36.071119 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954179 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:58:36.071119 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954182 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:58:36.071119 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954184 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:58:36.071119 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954187 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:58:36.074106 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954189 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:58:36.074106 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954192 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:58:36.074106 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954194 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:58:36.074106 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954197 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:58:36.074106 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954201 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:58:36.074106 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954206 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:58:36.074106 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954209 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:58:36.074106 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954212 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:58:36.074106 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954215 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:58:36.074106 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954217 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:58:36.074106 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954220 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:58:36.074106 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954223 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:58:36.074106 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954226 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:58:36.074106 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954228 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:58:36.074106 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954231 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:58:36.074106 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954233 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:58:36.074106 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954237 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:58:36.074106 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954239 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:58:36.074106 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954242 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:58:36.076201 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954246 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:58:36.076201 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954248 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:58:36.076201 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954251 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:58:36.076201 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954254 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:58:36.076201 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954256 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:58:36.076201 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954259 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:58:36.076201 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954261 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:58:36.076201 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954264 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:58:36.076201 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954267 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:58:36.076201 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954269 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:58:36.076201 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954272 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:58:36.076201 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954274 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:58:36.076201 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954277 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:58:36.076201 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954281 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:58:36.076201 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954284 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:58:36.076201 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954286 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:58:36.076201 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954289 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:58:36.076201 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954292 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:58:36.076201 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954295 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:58:36.076201 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954298 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:58:36.078997 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954300 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:58:36.078997 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954303 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:58:36.078997 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954305 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:58:36.078997 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954308 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:58:36.078997 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954310 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:58:36.078997 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954313 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:58:36.078997 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954316 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:58:36.078997 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954318 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:58:36.078997 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954321 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:58:36.078997 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954324 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:58:36.078997 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954326 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:58:36.078997 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954329 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:58:36.078997 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.954331 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:58:36.078997 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955090 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:58:36.078997 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955098 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:58:36.078997 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955101 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:58:36.078997 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955104 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:58:36.078997 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955108 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:58:36.078997 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955110 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:58:36.078997 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955114 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:58:36.092487 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955117 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:58:36.092487 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955120 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:58:36.092487 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955122 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:58:36.092487 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955125 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:58:36.092487 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955128 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:58:36.092487 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955131 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:58:36.092487 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955134 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:58:36.092487 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955137 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:58:36.092487 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955139 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:58:36.092487 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955142 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:58:36.092487 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955146 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:58:36.092487 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955149 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:58:36.092487 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955151 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:58:36.092487 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955154 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:58:36.092487 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955157 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:58:36.092487 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955159 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:58:36.092487 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955162 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:58:36.092487 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955165 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:58:36.092487 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955167 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:58:36.092487 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955170 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:58:36.094636 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955173 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:58:36.094636 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955176 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:58:36.094636 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955178 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:58:36.094636 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955181 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:58:36.094636 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955183 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:58:36.094636 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955186 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:58:36.094636 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955189 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:58:36.094636 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955192 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:58:36.094636 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955194 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:58:36.094636 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955197 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:58:36.094636 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955200 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:58:36.094636 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955203 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:58:36.094636 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955206 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:58:36.094636 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955208 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:58:36.094636 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955211 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:58:36.094636 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955213 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:58:36.094636 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955216 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:58:36.094636 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955219 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:58:36.094636 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955221 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:58:36.094636 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955224 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:58:36.094636 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955227 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:58:36.283655 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955230 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:58:36.283655 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955232 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:58:36.283655 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955236 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:58:36.283655 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955239 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:58:36.283655 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955241 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:58:36.283655 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955244 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:58:36.283655 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955246 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:58:36.283655 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955248 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:58:36.283655 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955251 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:58:36.283655 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955254 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:58:36.283655 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955257 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:58:36.283655 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955259 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:58:36.283655 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955262 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:58:36.283655 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955265 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:58:36.283655 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955267 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:58:36.283655 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955270 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:58:36.283655 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955272 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:58:36.283655 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955275 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:58:36.283655 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955277 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:58:36.283655 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955280 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:58:36.296258 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955283 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:58:36.296258 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955286 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:58:36.296258 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955289 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:58:36.296258 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955291 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:58:36.296258 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955296 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:58:36.296258 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955300 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:58:36.296258 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955303 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:58:36.296258 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955306 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:58:36.296258 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955308 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:58:36.296258 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955311 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:58:36.296258 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955314 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:58:36.296258 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955316 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:58:36.296258 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955319 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:58:36.296258 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955321 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:58:36.296258 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955324 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:58:36.296258 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955328 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:58:36.296258 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955331 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:58:36.296258 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955334 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:58:36.296258 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955338 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:58:36.297383 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955341 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:58:36.297383 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955344 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:58:36.297383 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955348 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:58:36.297383 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955350 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:58:36.297383 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955353 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:58:36.297383 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955356 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:58:36.297383 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955358 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:58:36.297383 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955362 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:58:36.297383 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955365 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:58:36.297383 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955368 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:58:36.297383 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955371 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:58:36.297383 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955373 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:58:36.297383 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955858 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:58:36.297383 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955864 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:58:36.297383 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955868 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:58:36.297383 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955872 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:58:36.297383 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955875 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:58:36.297383 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955878 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:58:36.297383 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955881 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:58:36.299209 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955884 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:58:36.299209 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955887 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:58:36.299209 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955890 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:58:36.299209 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955893 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:58:36.299209 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955896 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:58:36.299209 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955899 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:58:36.299209 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955901 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:58:36.299209 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955904 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:58:36.299209 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955907 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:58:36.299209 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955912 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:58:36.299209 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955916 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:58:36.299209 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955919 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:58:36.299209 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955922 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:58:36.299209 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955924 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:58:36.299209 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955927 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:58:36.299209 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955930 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:58:36.299209 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955934 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:58:36.299209 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955937 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:58:36.299209 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955941 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:58:36.302530 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955945 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:58:36.302530 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955948 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:58:36.302530 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955950 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:58:36.302530 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955953 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:58:36.302530 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955955 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:58:36.302530 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955958 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:58:36.302530 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955960 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:58:36.302530 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955963 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:58:36.302530 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955969 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:58:36.302530 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955974 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:58:36.302530 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955976 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:58:36.302530 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955979 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:58:36.302530 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955982 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:58:36.302530 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955984 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:58:36.302530 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955987 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:58:36.302530 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955990 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:58:36.302530 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955992 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:58:36.302530 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955995 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:58:36.302530 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.955999 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:58:36.302530 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956002 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:58:36.305286 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956022 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:58:36.305286 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956025 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:58:36.305286 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956027 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:58:36.305286 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956031 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:58:36.305286 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956033 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:58:36.305286 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956037 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:58:36.305286 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956039 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:58:36.305286 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956042 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:58:36.305286 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956045 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:58:36.305286 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956048 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:58:36.305286 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956050 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:58:36.305286 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956053 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:58:36.305286 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956056 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:58:36.305286 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956058 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:58:36.305286 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956061 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:58:36.305286 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956064 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:58:36.305286 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956066 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:58:36.305286 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956069 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:58:36.305286 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956071 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:58:36.308002 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956074 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:58:36.308002 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956077 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:58:36.308002 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956081 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:58:36.308002 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956084 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:58:36.308002 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956087 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:58:36.308002 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956089 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:58:36.308002 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956091 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:58:36.308002 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956099 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:58:36.308002 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956102 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:58:36.308002 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956105 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:58:36.308002 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956108 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:58:36.308002 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956110 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:58:36.308002 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956113 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:58:36.308002 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956115 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:58:36.308002 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956118 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:58:36.308002 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956120 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:58:36.308002 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956123 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:58:36.308002 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956126 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:58:36.308002 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956129 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:58:36.308002 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956132 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:58:36.308888 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956135 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:58:36.308888 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956137 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:58:36.308888 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956141 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:58:36.308888 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956145 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:58:36.308888 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956147 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:58:36.308888 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956150 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:58:36.308888 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956153 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:58:36.308888 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956156 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:58:36.308888 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956158 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:58:36.308888 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956161 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:58:36.308888 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956164 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:58:36.308888 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956166 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:58:36.308888 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956169 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:58:36.308888 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956171 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:58:36.308888 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956247 2577 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 20:58:36.308888 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956256 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 20:58:36.308888 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956263 2577 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 20:58:36.308888 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956268 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 20:58:36.308888 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956273 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 20:58:36.308888 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956276 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 20:58:36.311488 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956281 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 20:58:36.311488 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956286 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 20:58:36.311488 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956289 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 20:58:36.311488 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956292 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 20:58:36.311488 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956296 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 20:58:36.311488 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956299 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 20:58:36.311488 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956303 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 20:58:36.311488 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956306 2577 flags.go:64] FLAG: --cgroup-root="" Jul 06 20:58:36.311488 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956309 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 20:58:36.311488 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956312 2577 flags.go:64] FLAG: --client-ca-file="" Jul 06 20:58:36.311488 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956316 2577 flags.go:64] FLAG: --cloud-provider="external" Jul 06 20:58:36.311488 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956319 2577 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 20:58:36.311488 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956324 2577 flags.go:64] FLAG: --cluster-domain="" Jul 06 20:58:36.311488 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956327 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 20:58:36.311488 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956330 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 20:58:36.311488 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956333 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 20:58:36.311488 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956337 2577 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 20:58:36.311488 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956341 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 20:58:36.311488 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956344 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 20:58:36.311488 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956347 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 20:58:36.311488 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956350 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 20:58:36.311488 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956353 2577 flags.go:64] FLAG: --contention-profiling="false" Jul 06 20:58:36.311488 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956356 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 20:58:36.311488 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956359 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 20:58:36.312946 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956363 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 20:58:36.312946 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956366 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 20:58:36.312946 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956372 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 20:58:36.312946 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956375 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 20:58:36.312946 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956378 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 20:58:36.312946 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956383 2577 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 20:58:36.312946 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956386 2577 flags.go:64] FLAG: --enable-server="true" Jul 06 20:58:36.312946 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956389 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 20:58:36.312946 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956395 2577 flags.go:64] FLAG: --event-burst="100" Jul 06 20:58:36.312946 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956398 2577 flags.go:64] FLAG: --event-qps="50" Jul 06 20:58:36.312946 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956401 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 20:58:36.312946 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956404 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 20:58:36.312946 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956407 2577 flags.go:64] FLAG: --eviction-hard="" Jul 06 20:58:36.312946 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956411 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 20:58:36.312946 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956414 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 20:58:36.312946 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956418 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 20:58:36.312946 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956421 2577 flags.go:64] FLAG: --eviction-soft="" Jul 06 20:58:36.312946 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956424 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 20:58:36.312946 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956428 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 20:58:36.312946 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956431 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 20:58:36.312946 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956434 2577 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 20:58:36.312946 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956437 2577 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 20:58:36.312946 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956440 2577 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 20:58:36.312946 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956443 2577 flags.go:64] FLAG: --feature-gates="" Jul 06 20:58:36.312946 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956448 2577 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 20:58:36.313980 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956451 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 20:58:36.313980 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956454 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 20:58:36.313980 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956457 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 20:58:36.313980 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956460 2577 flags.go:64] FLAG: --healthz-port="10248" Jul 06 20:58:36.313980 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956463 2577 flags.go:64] FLAG: --help="false" Jul 06 20:58:36.313980 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956466 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-136-207.ec2.internal" Jul 06 20:58:36.313980 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956469 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 20:58:36.313980 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956472 2577 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 20:58:36.313980 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956475 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 20:58:36.313980 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956481 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 20:58:36.313980 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956484 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 20:58:36.313980 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956489 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 20:58:36.313980 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956495 2577 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 20:58:36.313980 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956499 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 20:58:36.313980 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956503 2577 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 20:58:36.313980 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956508 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 20:58:36.313980 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956512 2577 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 20:58:36.313980 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956515 2577 flags.go:64] FLAG: --kube-reserved="" Jul 06 20:58:36.313980 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956518 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 20:58:36.313980 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956521 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 20:58:36.313980 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956524 2577 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 20:58:36.313980 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956526 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 20:58:36.313980 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956529 2577 flags.go:64] FLAG: --lock-file="" Jul 06 20:58:36.313980 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956532 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 20:58:36.314917 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956535 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 20:58:36.314917 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956538 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 20:58:36.314917 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956544 2577 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 20:58:36.314917 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956554 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 20:58:36.314917 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956557 2577 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 20:58:36.314917 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956560 2577 flags.go:64] FLAG: --logging-format="text" Jul 06 20:58:36.314917 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956564 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 20:58:36.314917 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956567 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 20:58:36.314917 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956570 2577 flags.go:64] FLAG: --manifest-url="" Jul 06 20:58:36.314917 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956574 2577 flags.go:64] FLAG: --manifest-url-header="" Jul 06 20:58:36.314917 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956578 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 20:58:36.314917 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956581 2577 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 20:58:36.314917 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956589 2577 flags.go:64] FLAG: --max-pods="110" Jul 06 20:58:36.314917 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956592 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 20:58:36.314917 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956595 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 20:58:36.314917 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956598 2577 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 20:58:36.314917 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956601 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 20:58:36.314917 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956605 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 20:58:36.314917 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956608 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 20:58:36.314917 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956611 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 20:58:36.314917 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956621 2577 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 20:58:36.314917 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956624 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 20:58:36.314917 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956627 2577 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 20:58:36.314917 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956630 2577 flags.go:64] FLAG: --pod-cidr="" Jul 06 20:58:36.317754 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956632 2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 20:58:36.317754 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956638 2577 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 20:58:36.317754 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956641 2577 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 20:58:36.317754 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956645 2577 flags.go:64] FLAG: --pods-per-core="0" Jul 06 20:58:36.317754 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956648 2577 flags.go:64] FLAG: --port="10250" Jul 06 20:58:36.317754 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956650 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 20:58:36.317754 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956653 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-040062e752b8e3972" Jul 06 20:58:36.317754 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956657 2577 flags.go:64] FLAG: --qos-reserved="" Jul 06 20:58:36.317754 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956660 2577 flags.go:64] FLAG: --read-only-port="10255" Jul 06 20:58:36.317754 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956663 2577 flags.go:64] FLAG: --register-node="true" Jul 06 20:58:36.317754 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956665 2577 flags.go:64] FLAG: --register-with-taints="" Jul 06 20:58:36.317754 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956669 2577 flags.go:64] FLAG: --registry-burst="10" Jul 06 20:58:36.317754 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956673 2577 flags.go:64] FLAG: --registry-qps="5" Jul 06 20:58:36.317754 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956676 2577 flags.go:64] FLAG: --reserved-cpus="" Jul 06 20:58:36.317754 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956679 2577 flags.go:64] FLAG: --reserved-memory="" Jul 06 20:58:36.317754 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956682 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 20:58:36.317754 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956685 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 20:58:36.317754 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956688 2577 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 20:58:36.317754 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956691 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 20:58:36.317754 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956694 2577 flags.go:64] FLAG: --runonce="false" Jul 06 20:58:36.317754 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956696 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 20:58:36.317754 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956699 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 20:58:36.317754 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956702 2577 flags.go:64] FLAG: --seccomp-default="false" Jul 06 20:58:36.317754 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956705 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 20:58:36.317754 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956708 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 20:58:36.318652 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956713 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 20:58:36.318652 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956716 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 20:58:36.318652 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956719 2577 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 20:58:36.318652 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956722 2577 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 20:58:36.318652 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956726 2577 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 20:58:36.318652 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956729 2577 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 20:58:36.318652 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956732 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 20:58:36.318652 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956735 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 20:58:36.318652 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956738 2577 flags.go:64] FLAG: --system-cgroups="" Jul 06 20:58:36.318652 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956741 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 20:58:36.318652 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956746 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 20:58:36.318652 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956749 2577 flags.go:64] FLAG: --tls-cert-file="" Jul 06 20:58:36.318652 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956752 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 20:58:36.318652 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956757 2577 flags.go:64] FLAG: --tls-min-version="" Jul 06 20:58:36.318652 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956760 2577 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 20:58:36.318652 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956763 2577 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 20:58:36.318652 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956766 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 20:58:36.318652 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956769 2577 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 20:58:36.318652 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956772 2577 flags.go:64] FLAG: --v="2" Jul 06 20:58:36.318652 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956776 2577 flags.go:64] FLAG: --version="false" Jul 06 20:58:36.318652 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956786 2577 flags.go:64] FLAG: --vmodule="" Jul 06 20:58:36.318652 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956791 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 20:58:36.318652 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.956794 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 20:58:36.318652 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956922 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:58:36.318652 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956926 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:58:36.320018 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956929 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:58:36.320018 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956932 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:58:36.320018 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956935 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:58:36.320018 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956938 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:58:36.320018 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956942 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:58:36.320018 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956945 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:58:36.320018 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956948 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:58:36.320018 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956952 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:58:36.320018 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956955 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:58:36.320018 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956958 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:58:36.320018 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956961 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:58:36.320018 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956965 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:58:36.320018 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956968 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:58:36.320018 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956970 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:58:36.320018 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956973 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:58:36.320018 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956976 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:58:36.320018 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956978 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:58:36.320018 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956981 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:58:36.320018 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956984 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:58:36.320018 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956986 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:58:36.320698 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956989 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:58:36.320698 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956991 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:58:36.320698 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956994 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:58:36.320698 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956996 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:58:36.320698 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.956999 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:58:36.320698 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957002 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:58:36.320698 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957024 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:58:36.320698 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957027 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:58:36.320698 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957031 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:58:36.320698 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957034 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:58:36.320698 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957037 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:58:36.320698 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957041 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:58:36.320698 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957044 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:58:36.320698 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957047 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:58:36.320698 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957049 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:58:36.320698 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957052 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:58:36.320698 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957055 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:58:36.320698 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957057 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:58:36.320698 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957060 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:58:36.321603 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957064 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:58:36.321603 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957067 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:58:36.321603 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957070 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:58:36.321603 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957072 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:58:36.321603 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957076 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:58:36.321603 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957079 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:58:36.321603 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957082 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:58:36.321603 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957085 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:58:36.321603 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957087 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:58:36.321603 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957090 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:58:36.321603 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957093 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:58:36.321603 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957095 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:58:36.321603 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957098 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:58:36.321603 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957101 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:58:36.321603 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957103 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:58:36.321603 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957106 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:58:36.321603 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957108 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:58:36.321603 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957111 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:58:36.321603 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957114 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:58:36.321603 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957116 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:58:36.322310 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957119 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:58:36.322310 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957122 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:58:36.322310 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957128 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:58:36.322310 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957131 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:58:36.322310 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957135 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:58:36.322310 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957138 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:58:36.322310 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957141 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:58:36.322310 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957144 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:58:36.322310 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957147 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:58:36.322310 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957150 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:58:36.322310 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957152 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:58:36.322310 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957155 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:58:36.322310 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957159 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:58:36.322310 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957162 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:58:36.322310 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957164 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:58:36.322310 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957167 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:58:36.322310 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957170 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:58:36.322310 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957173 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:58:36.322310 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957176 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:58:36.326852 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957178 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:58:36.326852 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957181 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:58:36.326852 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957183 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:58:36.326852 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957186 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:58:36.326852 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957189 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:58:36.326852 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957191 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:58:36.326852 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957194 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:58:36.326852 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957196 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:58:36.326852 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957198 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:58:36.326852 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957201 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:58:36.326852 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957204 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:58:36.326852 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957207 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:58:36.326852 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957209 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:58:36.326852 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957212 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:58:36.326852 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957214 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:58:36.326852 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957217 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:58:36.326852 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957222 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:58:36.326852 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957225 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:58:36.326852 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.957228 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:58:36.327574 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.957940 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 20:58:36.327574 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.965802 2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 20:58:36.327574 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.965830 2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 20:58:36.327574 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.965852 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 20:58:36.327574 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.965858 2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 20:58:36.327574 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.965927 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:58:36.327574 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.965933 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:58:36.327574 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.965936 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:58:36.327574 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.965939 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:58:36.327574 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.965942 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:58:36.327574 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.965945 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:58:36.327574 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.965947 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:58:36.327574 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.965950 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:58:36.327574 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.965953 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:58:36.327574 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.965966 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:58:36.328725 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.965970 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:58:36.328725 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.965972 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:58:36.328725 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.965975 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:58:36.328725 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.965978 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:58:36.328725 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.965980 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:58:36.328725 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.965983 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:58:36.328725 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.965986 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:58:36.328725 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.965989 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:58:36.328725 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.965991 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:58:36.328725 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.965994 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:58:36.328725 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.965997 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:58:36.328725 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.965999 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:58:36.328725 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966002 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:58:36.328725 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966020 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:58:36.328725 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966024 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:58:36.328725 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966028 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:58:36.328725 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966031 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:58:36.328725 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966034 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:58:36.328725 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966037 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:58:36.328725 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966047 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:58:36.329415 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966050 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:58:36.329415 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966053 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:58:36.329415 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966056 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:58:36.329415 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966058 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:58:36.329415 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966061 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:58:36.329415 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966064 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:58:36.329415 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966066 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:58:36.329415 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966069 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:58:36.329415 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966071 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:58:36.329415 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966074 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:58:36.329415 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966076 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:58:36.329415 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966079 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:58:36.329415 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966082 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:58:36.329415 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966084 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:58:36.329415 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966087 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:58:36.329415 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966089 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:58:36.329415 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966092 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:58:36.329415 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966095 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:58:36.329415 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966097 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:58:36.329415 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966100 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:58:36.330551 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966102 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:58:36.330551 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966106 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:58:36.330551 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966111 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:58:36.330551 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966114 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:58:36.330551 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966117 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:58:36.330551 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966120 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:58:36.330551 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966123 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:58:36.330551 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966126 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:58:36.330551 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966128 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:58:36.330551 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966131 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:58:36.330551 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966133 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:58:36.330551 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966137 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:58:36.330551 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966146 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:58:36.330551 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966149 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:58:36.330551 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966152 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:58:36.330551 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966154 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:58:36.330551 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966157 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:58:36.330551 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966159 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:58:36.330551 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966162 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:58:36.330551 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966164 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:58:36.390369 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966167 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:58:36.390369 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966170 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:58:36.390369 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966173 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:58:36.390369 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966175 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:58:36.390369 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966178 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:58:36.390369 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966180 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:58:36.390369 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966182 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:58:36.390369 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966185 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:58:36.390369 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966188 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:58:36.390369 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966190 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:58:36.390369 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966193 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:58:36.390369 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966195 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:58:36.390369 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966198 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:58:36.390369 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966200 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:58:36.390369 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966204 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:58:36.390369 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966208 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:58:36.390369 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966211 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:58:36.390369 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966214 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:58:36.390369 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966217 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:58:36.391283 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966219 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:58:36.391283 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966222 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:58:36.391283 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966225 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:58:36.391283 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966229 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:58:36.391283 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966234 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:58:36.391283 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966237 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:58:36.391283 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966245 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:58:36.391283 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966248 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:58:36.391283 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966251 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:58:36.391283 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966253 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:58:36.391283 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.966259 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 20:58:36.391283 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966426 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 20:58:36.391283 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966431 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 06 20:58:36.391283 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966434 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 20:58:36.391733 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966437 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 20:58:36.391733 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966440 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 20:58:36.391733 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966442 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 20:58:36.391733 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966445 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 20:58:36.391733 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966448 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 20:58:36.391733 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966451 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 20:58:36.391733 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966454 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 20:58:36.391733 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966456 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 20:58:36.391733 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966459 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 20:58:36.391733 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966482 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 20:58:36.391733 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966486 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 20:58:36.391733 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966489 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 20:58:36.391733 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966491 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 20:58:36.391733 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966495 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 20:58:36.391733 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966499 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 20:58:36.391733 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966501 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 20:58:36.391733 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966504 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 20:58:36.391733 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966511 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 20:58:36.391733 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966513 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 20:58:36.393334 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966516 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 20:58:36.393334 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966518 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 20:58:36.393334 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966521 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 20:58:36.393334 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966525 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 20:58:36.393334 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966527 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 20:58:36.393334 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966530 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 20:58:36.393334 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966540 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 20:58:36.393334 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966543 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 20:58:36.393334 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966546 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 20:58:36.393334 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966548 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 20:58:36.393334 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966551 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 20:58:36.393334 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966553 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 20:58:36.393334 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966556 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 20:58:36.393334 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966559 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 20:58:36.393334 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966561 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 20:58:36.393334 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966565 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 20:58:36.393334 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966568 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 20:58:36.393334 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966571 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 20:58:36.393334 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966573 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 20:58:36.393334 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966576 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 20:58:36.394033 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966579 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 20:58:36.394033 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966581 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 20:58:36.394033 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966584 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 20:58:36.394033 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966586 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 20:58:36.394033 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966589 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 20:58:36.394033 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966591 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 20:58:36.394033 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966594 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 20:58:36.394033 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966596 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 20:58:36.394033 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966599 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 20:58:36.394033 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966601 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 20:58:36.394033 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966606 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 20:58:36.394033 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966608 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 20:58:36.394033 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966611 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 20:58:36.394033 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966614 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 20:58:36.394033 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966616 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 20:58:36.394033 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966620 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 20:58:36.394033 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966623 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 20:58:36.394033 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966626 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 20:58:36.394033 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966628 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 20:58:36.394033 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966637 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 20:58:36.394711 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966640 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 20:58:36.394711 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966642 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 20:58:36.394711 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966645 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 20:58:36.394711 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966647 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 20:58:36.394711 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966650 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 20:58:36.394711 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966653 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 20:58:36.394711 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966655 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 20:58:36.394711 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966657 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 20:58:36.394711 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966660 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 20:58:36.394711 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966662 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 20:58:36.394711 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966665 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 20:58:36.394711 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966667 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 20:58:36.394711 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966670 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 20:58:36.394711 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966672 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 20:58:36.394711 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966675 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 20:58:36.394711 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966677 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 20:58:36.394711 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966680 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 20:58:36.394711 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966682 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 20:58:36.394711 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966686 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 20:58:36.396535 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966690 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 20:58:36.396535 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966692 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 20:58:36.396535 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966695 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 20:58:36.396535 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966699 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 20:58:36.396535 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966702 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 20:58:36.396535 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966704 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 20:58:36.396535 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966707 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 20:58:36.396535 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966710 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 20:58:36.396535 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966714 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 20:58:36.396535 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966716 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 20:58:36.396535 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966719 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 20:58:36.396535 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966721 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 20:58:36.396535 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966724 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 20:58:36.396535 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966732 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 20:58:36.396535 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966735 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 20:58:36.396535 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966738 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 20:58:36.396535 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966740 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 20:58:36.396535 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:35.966743 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 20:58:36.397221 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.966747 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 20:58:36.397221 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.966913 2577 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 20:58:36.397221 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.971552 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 20:58:36.397221 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.973147 2577 server.go:1019] "Starting client certificate rotation" Jul 06 20:58:36.397221 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.973256 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 20:58:36.397221 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.973310 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 20:58:36.397221 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.976135 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 20:58:36.397221 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.983586 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 20:58:36.397221 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.995675 2577 log.go:25] "Validated CRI v1 runtime API" Jul 06 20:58:36.397221 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.998093 2577 log.go:25] "Validated CRI v1 image API" Jul 06 20:58:36.397221 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:35.998910 2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 20:58:36.397221 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.001461 2577 fs.go:135] Filesystem UUIDs: map[4dc792df-a81b-4fff-989b-dd21a0e58b1e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7f75637a-871e-4e71-857b-7ad069a7bb82:/dev/nvme0n1p4] Jul 06 20:58:36.398599 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.001477 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 20:58:36.398833 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.007704 2577 manager.go:216] Machine: {Timestamp:2026-07-06 20:58:36.005932871 +0000 UTC m=+0.324156123 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099676 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29fcc1bcb798039d5e996aab0656dd SystemUUID:ec29fcc1-bcb7-9803-9d5e-996aab0656dd BootID:7ea7ee88-7713-4db0-a97d-ec404bf3dba4 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1c:16:37:08:3b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1c:16:37:08:3b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:c6:45:3f:36:15 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 20:58:36.398833 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.007931 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 20:58:36.398833 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.008101 2577 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 20:58:36.398833 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.010912 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 20:58:36.398833 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.011795 2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 20:58:36.398833 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.011825 2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-207.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 20:58:36.398833 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.013386 2577 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.013397 2577 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.013406 2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.014384 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.016233 2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.019592 2577 state_mem.go:36] "Initialized new in-memory state store" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.019751 2577 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.019786 2577 kubelet.go:486] "Attempting to sync node with API server" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.019796 2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.019813 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.019822 2577 kubelet.go:398] "Adding apiserver pod source" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.019832 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.020294 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.020310 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.022915 2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.026219 2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.026273 2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.027674 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.027690 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.027696 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.027702 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.027711 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.027717 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.027726 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.027734 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.027779 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.027786 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.027801 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.027807 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.028712 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.028726 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.031861 2577 server.go:1268] "Started kubelet" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.031966 2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.031945 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.032191 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:36.036266 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-207.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.036266 2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:36.036532 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.037539 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-207.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.037796 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7hbwd" Jul 06 20:58:36.399993 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.038019 2577 server.go:310] "Adding debug handlers to kubelet server" Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:36.042731 2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.044229 2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.044239 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:36.044342 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-207.ec2.internal\" not found" Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.044999 2577 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.045027 2577 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.045447 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:36.042420 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-207.ec2.internal.18bfcef1f3a08d02 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-207.ec2.internal,UID:ip-10-0-136-207.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-207.ec2.internal,},FirstTimestamp:2026-07-06 20:58:36.031831298 +0000 UTC m=+0.350054548,LastTimestamp:2026-07-06 20:58:36.031831298 +0000 UTC m=+0.350054548,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-207.ec2.internal,}" Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.045680 2577 reconstruct.go:97] "Volume reconstruction finished" Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.045691 2577 reconciler.go:29] "Reconciler: start to sync state" Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.045810 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.045826 2577 factory.go:55] Registering systemd factory Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.045836 2577 factory.go:223] Registration of the systemd container factory successfully Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.045838 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7hbwd" Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.046072 2577 factory.go:153] Registering CRI-O factory Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.046087 2577 factory.go:223] Registration of the crio container factory successfully Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.046110 2577 factory.go:103] Registering Raw factory Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.046131 2577 manager.go:1196] Started watching for new ooms in manager Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.046422 2577 manager.go:349] Starting recovery of all containers Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.052565 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.058820 2577 manager.go:354] Recovery completed Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.060756 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:36.066286 2577 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-207.ec2.internal\" not found" node="ip-10-0-136-207.ec2.internal" Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.066787 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.069751 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.069786 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.069806 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.070425 2577 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.070435 2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.070440 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.070457 2577 state_mem.go:36] "Initialized new in-memory state store" Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.071419 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.071440 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.071454 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.072702 2577 policy_none.go:49] "None policy: Start" Jul 06 20:58:36.401157 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.072717 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.072727 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.073705 2577 policy_none.go:47] "Start" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.114747 2577 manager.go:337] "Starting Device Plugin manager" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:36.114827 2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.114841 2577 server.go:87] "Starting device plugin registration server" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.115133 2577 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.115144 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.115245 2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.115507 2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.115517 2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:36.115833 2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:36.115884 2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-207.ec2.internal\" not found" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.161987 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.162042 2577 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.162070 2577 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:36.162108 2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.164656 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.216259 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.217331 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.217362 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.217375 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.217405 2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-207.ec2.internal" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.227385 2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-207.ec2.internal" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:36.227410 2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-207.ec2.internal\": node \"ip-10-0-136-207.ec2.internal\" not found" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:36.249411 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-207.ec2.internal\" not found" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.262371 2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-207.ec2.internal"] Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.262479 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.263482 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.263514 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.263524 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.264822 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.264932 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.264962 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.265649 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.265674 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.265686 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:58:36.402310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.265649 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:58:36.403454 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.265762 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:58:36.403454 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.265781 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:58:36.403454 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.266866 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-207.ec2.internal" Jul 06 20:58:36.403454 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.266890 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 20:58:36.403454 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.267805 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasSufficientMemory" Jul 06 20:58:36.403454 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.267833 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 20:58:36.403454 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.267847 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeHasSufficientPID" Jul 06 20:58:36.403454 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:36.288744 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-207.ec2.internal\" not found" node="ip-10-0-136-207.ec2.internal" Jul 06 20:58:36.403454 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:36.292385 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-207.ec2.internal\" not found" node="ip-10-0-136-207.ec2.internal" Jul 06 20:58:36.985709 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.985662 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 20:58:36.985709 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:36.985684 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-207.ec2.internal\" not found" Jul 06 20:58:36.985933 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.985793 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/90a399fb4023b43548b92e7e89de6a41-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal\" (UID: \"90a399fb4023b43548b92e7e89de6a41\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal" Jul 06 20:58:36.985933 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.985874 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 20:58:36.985933 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.985874 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 20:58:36.998864 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:36.998833 2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:58:37.020916 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.020890 2577 apiserver.go:52] "Watching apiserver" Jul 06 20:58:37.030422 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.030391 2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 20:58:37.032414 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.032379 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-d829k","openshift-multus/network-metrics-daemon-qc5r5","kube-system/global-pull-secret-syncer-dbnt9","openshift-network-diagnostics/network-check-target-n8x9c","openshift-network-operator/iptables-alerter-vgct2","openshift-ovn-kubernetes/ovnkube-node-qjs2n","kube-system/konnectivity-agent-pvvfl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntx72","openshift-cluster-node-tuning-operator/tuned-8bxzl","openshift-dns/node-resolver-9mn4b","openshift-image-registry/node-ca-dmgkn","openshift-multus/multus-additional-cni-plugins-v7sdw"] Jul 06 20:58:37.034238 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.034216 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d829k" Jul 06 20:58:37.035566 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.035535 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:58:37.035704 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:37.035683 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc5r5" podUID="6bb28f09-af91-446b-9f07-6a9a331b2bf5" Jul 06 20:58:37.036902 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.036877 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:58:37.037008 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:37.036963 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbnt9" podUID="b0f9a2ff-6f72-4374-a459-b4da598d3533" Jul 06 20:58:37.037076 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.037055 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 20:58:37.037155 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.037081 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 20:58:37.037155 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.037064 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 20:58:37.037155 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.037109 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-f59z7\"" Jul 06 20:58:37.037274 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.037104 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 20:58:37.039222 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.039178 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:58:37.039364 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:37.039248 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8x9c" podUID="b3810ec4-264c-4e94-ba9e-ab262e657770" Jul 06 20:58:37.042356 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.042327 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vgct2" Jul 06 20:58:37.043729 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.043710 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.044962 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.044937 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal" Jul 06 20:58:37.045129 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.045083 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pvvfl" Jul 06 20:58:37.045257 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.044942 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 20:58:37.046546 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.046525 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntx72" Jul 06 20:58:37.047054 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.047033 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 20:58:37.047333 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.047310 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 20:58:37.047445 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.047342 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 20:58:37.047445 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.047366 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 20:58:37.047445 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.047343 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9gwdv\"" Jul 06 20:58:37.047680 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.047661 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 20:58:37.047762 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.047689 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 20:58:37.047762 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.047697 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 20:58:37.047762 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.047692 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c7xhd\"" Jul 06 20:58:37.047995 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.047977 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 20:58:37.048239 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.048220 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.049330 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.049314 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 20:58:37.049421 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.049404 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 20:58:37.049510 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.049465 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 20:53:36 +0000 UTC" deadline="2028-01-05 21:22:45.76054003 +0000 UTC" Jul 06 20:58:37.049564 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.049513 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13152h24m8.711031467s" Jul 06 20:58:37.049640 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.049623 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 20:58:37.049707 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.049651 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 20:58:37.049775 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.049753 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dcz7n\"" Jul 06 20:58:37.049829 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.049815 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2ctjn\"" Jul 06 20:58:37.049917 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.049899 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 20:58:37.050010 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.049941 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9mn4b" Jul 06 20:58:37.050010 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.049954 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 20:58:37.052483 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.052466 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dmgkn" Jul 06 20:58:37.054063 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.054040 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v7sdw" Jul 06 20:58:37.055930 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.055903 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 20:58:37.056101 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.055940 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 20:58:37.056185 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.056163 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 20:58:37.056281 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.056184 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 20:58:37.056281 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.056200 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 20:58:37.056281 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.056248 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-svhj9\"" Jul 06 20:58:37.056445 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.056171 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 20:58:37.056445 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.056310 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 20:58:37.056445 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.056319 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q99fj\"" Jul 06 20:58:37.056445 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.056263 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 20:58:37.056664 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.056648 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kptx6\"" Jul 06 20:58:37.056664 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.056662 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j9ztx\"" Jul 06 20:58:37.056729 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.056648 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 20:58:37.061159 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.061135 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 20:58:37.061326 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.061236 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-207.ec2.internal" Jul 06 20:58:37.061519 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.061494 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal"] Jul 06 20:58:37.064070 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.064044 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 20:58:37.071070 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.071041 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-207.ec2.internal"] Jul 06 20:58:37.071596 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.071085 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 20:58:37.081255 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.081228 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7t5b4" Jul 06 20:58:37.086434 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.086397 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-host-kubelet\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.086434 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.086437 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-node-log\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.086619 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.086464 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90a399fb4023b43548b92e7e89de6a41-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal\" (UID: \"90a399fb4023b43548b92e7e89de6a41\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal" Jul 06 20:58:37.086619 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.086494 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ntx72\" (UID: \"6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntx72" Jul 06 20:58:37.086619 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.086519 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-cnibin\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.086619 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.086535 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aec1284f-edc5-472a-9552-2d73b756df48-cni-binary-copy\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.086619 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.086553 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-host-var-lib-kubelet\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.086619 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.086573 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-host-run-netns\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.086619 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.086591 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/835fe22bba349376f78a29c5926e82d5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-207.ec2.internal\" (UID: \"835fe22bba349376f78a29c5926e82d5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-207.ec2.internal" Jul 06 20:58:37.086887 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.086645 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8ca96cb1-d98e-4362-b6ad-00dd336bd23d-konnectivity-ca\") pod \"konnectivity-agent-pvvfl\" (UID: \"8ca96cb1-d98e-4362-b6ad-00dd336bd23d\") " pod="kube-system/konnectivity-agent-pvvfl" Jul 06 20:58:37.086887 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.086676 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-run-openvswitch\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.086887 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.086695 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b0f9a2ff-6f72-4374-a459-b4da598d3533-dbus\") pod \"global-pull-secret-syncer-dbnt9\" (UID: \"b0f9a2ff-6f72-4374-a459-b4da598d3533\") " pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:58:37.086887 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.086717 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa-registration-dir\") pod \"aws-ebs-csi-driver-node-ntx72\" (UID: \"6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntx72" Jul 06 20:58:37.086887 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.086735 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1d7b0d2-1555-4c35-a593-d96915d42afe-host-slash\") pod \"iptables-alerter-vgct2\" (UID: \"a1d7b0d2-1555-4c35-a593-d96915d42afe\") " pod="openshift-network-operator/iptables-alerter-vgct2" Jul 06 20:58:37.086887 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.086750 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-systemd-units\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.086887 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.086786 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-host-run-netns\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.086887 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.086852 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgv57\" (UniqueName: \"kubernetes.io/projected/6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa-kube-api-access-xgv57\") pod \"aws-ebs-csi-driver-node-ntx72\" (UID: \"6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntx72" Jul 06 20:58:37.087301 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.086894 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-host-var-lib-cni-bin\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.087301 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.086923 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bb28f09-af91-446b-9f07-6a9a331b2bf5-metrics-certs\") pod \"network-metrics-daemon-qc5r5\" (UID: \"6bb28f09-af91-446b-9f07-6a9a331b2bf5\") " pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:58:37.087301 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.086948 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a1d7b0d2-1555-4c35-a593-d96915d42afe-iptables-alerter-script\") pod \"iptables-alerter-vgct2\" (UID: \"a1d7b0d2-1555-4c35-a593-d96915d42afe\") " pod="openshift-network-operator/iptables-alerter-vgct2" Jul 06 20:58:37.087301 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.086974 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-log-socket\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.087301 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087017 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/90a399fb4023b43548b92e7e89de6a41-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal\" (UID: \"90a399fb4023b43548b92e7e89de6a41\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal" Jul 06 20:58:37.087301 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087071 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-run-systemd\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.087301 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087091 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/90a399fb4023b43548b92e7e89de6a41-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal\" (UID: \"90a399fb4023b43548b92e7e89de6a41\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal" Jul 06 20:58:37.087301 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087100 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.087301 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087128 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/64b45669-ed56-41b9-a3ed-1fe5c13508b6-ovn-node-metrics-cert\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.087301 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087153 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/64b45669-ed56-41b9-a3ed-1fe5c13508b6-env-overrides\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.087301 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087178 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0f9a2ff-6f72-4374-a459-b4da598d3533-original-pull-secret\") pod \"global-pull-secret-syncer-dbnt9\" (UID: \"b0f9a2ff-6f72-4374-a459-b4da598d3533\") " pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:58:37.087301 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087203 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aec1284f-edc5-472a-9552-2d73b756df48-multus-daemon-config\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.087301 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087227 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-etc-kubernetes\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.087943 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087255 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj6kq\" (UniqueName: \"kubernetes.io/projected/b3810ec4-264c-4e94-ba9e-ab262e657770-kube-api-access-lj6kq\") pod \"network-check-target-n8x9c\" (UID: \"b3810ec4-264c-4e94-ba9e-ab262e657770\") " pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:58:37.087943 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087276 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-var-lib-openvswitch\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.087943 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087302 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa-etc-selinux\") pod \"aws-ebs-csi-driver-node-ntx72\" (UID: \"6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntx72" Jul 06 20:58:37.087943 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087332 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-hostroot\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.087943 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087371 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxbrg\" (UniqueName: \"kubernetes.io/projected/6bb28f09-af91-446b-9f07-6a9a331b2bf5-kube-api-access-hxbrg\") pod \"network-metrics-daemon-qc5r5\" (UID: \"6bb28f09-af91-446b-9f07-6a9a331b2bf5\") " pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:58:37.087943 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087400 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-host-run-ovn-kubernetes\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.087943 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087427 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/64b45669-ed56-41b9-a3ed-1fe5c13508b6-ovnkube-config\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.087943 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087466 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa-sys-fs\") pod \"aws-ebs-csi-driver-node-ntx72\" (UID: \"6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntx72" Jul 06 20:58:37.087943 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087488 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-multus-socket-dir-parent\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.087943 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087506 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-host-run-multus-certs\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.087943 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087523 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-run-ovn\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.087943 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087548 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmh9x\" (UniqueName: \"kubernetes.io/projected/64b45669-ed56-41b9-a3ed-1fe5c13508b6-kube-api-access-mmh9x\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.087943 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087606 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8ca96cb1-d98e-4362-b6ad-00dd336bd23d-agent-certs\") pod \"konnectivity-agent-pvvfl\" (UID: \"8ca96cb1-d98e-4362-b6ad-00dd336bd23d\") " pod="kube-system/konnectivity-agent-pvvfl" Jul 06 20:58:37.087943 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087644 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-system-cni-dir\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.088400 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087671 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-host-run-k8s-cni-cncf-io\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.088400 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087721 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa-socket-dir\") pod \"aws-ebs-csi-driver-node-ntx72\" (UID: \"6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntx72" Jul 06 20:58:37.088400 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087748 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-os-release\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.088400 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087773 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-host-slash\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.088400 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087800 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-etc-openvswitch\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.088400 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087825 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/64b45669-ed56-41b9-a3ed-1fe5c13508b6-ovnkube-script-lib\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.088400 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087851 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tnpk\" (UniqueName: \"kubernetes.io/projected/a1d7b0d2-1555-4c35-a593-d96915d42afe-kube-api-access-4tnpk\") pod \"iptables-alerter-vgct2\" (UID: \"a1d7b0d2-1555-4c35-a593-d96915d42afe\") " pod="openshift-network-operator/iptables-alerter-vgct2" Jul 06 20:58:37.088400 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087876 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-multus-conf-dir\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.088400 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087899 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b0f9a2ff-6f72-4374-a459-b4da598d3533-kubelet-config\") pod \"global-pull-secret-syncer-dbnt9\" (UID: \"b0f9a2ff-6f72-4374-a459-b4da598d3533\") " pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:58:37.088400 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087925 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-multus-cni-dir\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.088400 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087956 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-host-var-lib-cni-multus\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.088400 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.087986 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-host-cni-netd\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.088400 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.088008 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kblxn\" (UniqueName: \"kubernetes.io/projected/aec1284f-edc5-472a-9552-2d73b756df48-kube-api-access-kblxn\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.088400 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.088045 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-host-cni-bin\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.088782 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.088060 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa-device-dir\") pod \"aws-ebs-csi-driver-node-ntx72\" (UID: \"6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntx72" Jul 06 20:58:37.092513 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.092485 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7t5b4" Jul 06 20:58:37.147035 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.147001 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 20:58:37.171775 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.171752 2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:58:37.188303 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.188279 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa-sys-fs\") pod \"aws-ebs-csi-driver-node-ntx72\" (UID: \"6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntx72" Jul 06 20:58:37.188432 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.188335 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-multus-socket-dir-parent\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.188432 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.188362 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-host-run-multus-certs\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.188432 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.188386 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-run-ovn\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.188432 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.188413 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmh9x\" (UniqueName: \"kubernetes.io/projected/64b45669-ed56-41b9-a3ed-1fe5c13508b6-kube-api-access-mmh9x\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.188638 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.188439 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa-sys-fs\") pod \"aws-ebs-csi-driver-node-ntx72\" (UID: \"6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntx72" Jul 06 20:58:37.188638 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.188451 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-etc-sysctl-conf\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.188638 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.188482 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-multus-socket-dir-parent\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.188638 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.188490 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8ca96cb1-d98e-4362-b6ad-00dd336bd23d-agent-certs\") pod \"konnectivity-agent-pvvfl\" (UID: \"8ca96cb1-d98e-4362-b6ad-00dd336bd23d\") " pod="kube-system/konnectivity-agent-pvvfl" Jul 06 20:58:37.188638 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.188512 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-run-ovn\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.188638 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.188522 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-system-cni-dir\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.188638 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.188543 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-host-run-k8s-cni-cncf-io\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.188638 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.188564 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-etc-sysconfig\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.188638 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.188574 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-host-run-multus-certs\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.188638 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.188590 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/006c0aa4-b1d8-4e60-af3d-6f994de5c237-tmp\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.188638 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.188616 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-system-cni-dir\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.188638 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.188625 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-host-run-k8s-cni-cncf-io\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.188638 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.188634 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa-socket-dir\") pod \"aws-ebs-csi-driver-node-ntx72\" (UID: \"6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntx72" Jul 06 20:58:37.189209 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.188655 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-os-release\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.189209 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.188693 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-host-slash\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.189209 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.188740 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-etc-openvswitch\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.189209 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.188754 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-os-release\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.189209 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.188787 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-host-slash\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.189209 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.188816 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/64b45669-ed56-41b9-a3ed-1fe5c13508b6-ovnkube-script-lib\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.189209 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.188845 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/826779c0-bcd6-4c50-9ec1-0b62c9122657-tmp-dir\") pod \"node-resolver-9mn4b\" (UID: \"826779c0-bcd6-4c50-9ec1-0b62c9122657\") " pod="openshift-dns/node-resolver-9mn4b" Jul 06 20:58:37.189209 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.188859 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa-socket-dir\") pod \"aws-ebs-csi-driver-node-ntx72\" (UID: \"6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntx72" Jul 06 20:58:37.189209 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.188883 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a9511f0e-6c1f-431c-8419-c1adc71f5520-serviceca\") pod \"node-ca-dmgkn\" (UID: \"a9511f0e-6c1f-431c-8419-c1adc71f5520\") " pod="openshift-image-registry/node-ca-dmgkn" Jul 06 20:58:37.189209 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.188910 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fcb1b646-77b2-4499-a4c7-4cfabfe3dcca-system-cni-dir\") pod \"multus-additional-cni-plugins-v7sdw\" (UID: \"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca\") " pod="openshift-multus/multus-additional-cni-plugins-v7sdw" Jul 06 20:58:37.189209 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.188938 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tnpk\" (UniqueName: \"kubernetes.io/projected/a1d7b0d2-1555-4c35-a593-d96915d42afe-kube-api-access-4tnpk\") pod \"iptables-alerter-vgct2\" (UID: \"a1d7b0d2-1555-4c35-a593-d96915d42afe\") " pod="openshift-network-operator/iptables-alerter-vgct2" Jul 06 20:58:37.189209 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.188936 2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 20:58:37.189209 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.188963 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-etc-systemd\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.189209 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.189021 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-run\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.189209 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.189048 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-var-lib-kubelet\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.190032 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.189095 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-host\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.190032 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.189146 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-multus-conf-dir\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.190032 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.189173 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b0f9a2ff-6f72-4374-a459-b4da598d3533-kubelet-config\") pod \"global-pull-secret-syncer-dbnt9\" (UID: \"b0f9a2ff-6f72-4374-a459-b4da598d3533\") " pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:58:37.190032 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.189199 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-multus-cni-dir\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.190032 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.189262 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-multus-cni-dir\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.190032 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.189321 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-etc-openvswitch\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.190032 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.189340 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-multus-conf-dir\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.190032 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.189369 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-host-var-lib-cni-multus\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.190032 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.189388 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b0f9a2ff-6f72-4374-a459-b4da598d3533-kubelet-config\") pod \"global-pull-secret-syncer-dbnt9\" (UID: \"b0f9a2ff-6f72-4374-a459-b4da598d3533\") " pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:58:37.190032 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.189404 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-host-cni-netd\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.190032 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.189460 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fcb1b646-77b2-4499-a4c7-4cfabfe3dcca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v7sdw\" (UID: \"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca\") " pod="openshift-multus/multus-additional-cni-plugins-v7sdw" Jul 06 20:58:37.190032 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.189459 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-host-var-lib-cni-multus\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.190032 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.189471 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-host-cni-netd\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.190032 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.189510 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kblxn\" (UniqueName: \"kubernetes.io/projected/aec1284f-edc5-472a-9552-2d73b756df48-kube-api-access-kblxn\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.190032 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.189543 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-host-cni-bin\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.190558 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.189624 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fg74\" (UniqueName: \"kubernetes.io/projected/a9511f0e-6c1f-431c-8419-c1adc71f5520-kube-api-access-7fg74\") pod \"node-ca-dmgkn\" (UID: \"a9511f0e-6c1f-431c-8419-c1adc71f5520\") " pod="openshift-image-registry/node-ca-dmgkn" Jul 06 20:58:37.190558 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.189634 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-host-cni-bin\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.190558 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.189646 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/64b45669-ed56-41b9-a3ed-1fe5c13508b6-ovnkube-script-lib\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.190558 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.189740 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa-device-dir\") pod \"aws-ebs-csi-driver-node-ntx72\" (UID: \"6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntx72" Jul 06 20:58:37.190558 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.189820 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-host-kubelet\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.190558 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.189810 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa-device-dir\") pod \"aws-ebs-csi-driver-node-ntx72\" (UID: \"6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntx72" Jul 06 20:58:37.190558 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.189881 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-host-kubelet\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.190558 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190039 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-node-log\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.190558 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.189869 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-node-log\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.190558 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190321 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjchr\" (UniqueName: \"kubernetes.io/projected/826779c0-bcd6-4c50-9ec1-0b62c9122657-kube-api-access-hjchr\") pod \"node-resolver-9mn4b\" (UID: \"826779c0-bcd6-4c50-9ec1-0b62c9122657\") " pod="openshift-dns/node-resolver-9mn4b" Jul 06 20:58:37.190558 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190339 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-etc-sysctl-d\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.190558 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190357 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90a399fb4023b43548b92e7e89de6a41-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal\" (UID: \"90a399fb4023b43548b92e7e89de6a41\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal" Jul 06 20:58:37.190558 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190374 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ntx72\" (UID: \"6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntx72" Jul 06 20:58:37.190558 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190390 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-cnibin\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.191144 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190407 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aec1284f-edc5-472a-9552-2d73b756df48-cni-binary-copy\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.191144 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190425 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-host-var-lib-kubelet\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.191144 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190445 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-cnibin\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.191144 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190452 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fcb1b646-77b2-4499-a4c7-4cfabfe3dcca-cni-binary-copy\") pod \"multus-additional-cni-plugins-v7sdw\" (UID: \"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca\") " pod="openshift-multus/multus-additional-cni-plugins-v7sdw" Jul 06 20:58:37.191144 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190456 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90a399fb4023b43548b92e7e89de6a41-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal\" (UID: \"90a399fb4023b43548b92e7e89de6a41\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal" Jul 06 20:58:37.191144 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190458 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ntx72\" (UID: \"6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntx72" Jul 06 20:58:37.191144 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190486 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fcb1b646-77b2-4499-a4c7-4cfabfe3dcca-os-release\") pod \"multus-additional-cni-plugins-v7sdw\" (UID: \"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca\") " pod="openshift-multus/multus-additional-cni-plugins-v7sdw" Jul 06 20:58:37.191144 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190494 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-host-var-lib-kubelet\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.191144 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190537 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fcb1b646-77b2-4499-a4c7-4cfabfe3dcca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v7sdw\" (UID: \"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca\") " pod="openshift-multus/multus-additional-cni-plugins-v7sdw" Jul 06 20:58:37.191144 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190564 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-host-run-netns\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.191144 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190597 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fcb1b646-77b2-4499-a4c7-4cfabfe3dcca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v7sdw\" (UID: \"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca\") " pod="openshift-multus/multus-additional-cni-plugins-v7sdw" Jul 06 20:58:37.191144 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190608 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-host-run-netns\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.191144 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190635 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/835fe22bba349376f78a29c5926e82d5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-207.ec2.internal\" (UID: \"835fe22bba349376f78a29c5926e82d5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-207.ec2.internal" Jul 06 20:58:37.191144 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190664 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8ca96cb1-d98e-4362-b6ad-00dd336bd23d-konnectivity-ca\") pod \"konnectivity-agent-pvvfl\" (UID: \"8ca96cb1-d98e-4362-b6ad-00dd336bd23d\") " pod="kube-system/konnectivity-agent-pvvfl" Jul 06 20:58:37.191815 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190674 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/835fe22bba349376f78a29c5926e82d5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-207.ec2.internal\" (UID: \"835fe22bba349376f78a29c5926e82d5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-207.ec2.internal" Jul 06 20:58:37.191815 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190713 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-run-openvswitch\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.191815 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190746 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/826779c0-bcd6-4c50-9ec1-0b62c9122657-hosts-file\") pod \"node-resolver-9mn4b\" (UID: \"826779c0-bcd6-4c50-9ec1-0b62c9122657\") " pod="openshift-dns/node-resolver-9mn4b" Jul 06 20:58:37.191815 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190754 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-run-openvswitch\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.191815 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190809 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b0f9a2ff-6f72-4374-a459-b4da598d3533-dbus\") pod \"global-pull-secret-syncer-dbnt9\" (UID: \"b0f9a2ff-6f72-4374-a459-b4da598d3533\") " pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:58:37.191815 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190839 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa-registration-dir\") pod \"aws-ebs-csi-driver-node-ntx72\" (UID: \"6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntx72" Jul 06 20:58:37.191815 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190866 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1d7b0d2-1555-4c35-a593-d96915d42afe-host-slash\") pod \"iptables-alerter-vgct2\" (UID: \"a1d7b0d2-1555-4c35-a593-d96915d42afe\") " pod="openshift-network-operator/iptables-alerter-vgct2" Jul 06 20:58:37.191815 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190890 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b0f9a2ff-6f72-4374-a459-b4da598d3533-dbus\") pod \"global-pull-secret-syncer-dbnt9\" (UID: \"b0f9a2ff-6f72-4374-a459-b4da598d3533\") " pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:58:37.191815 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190893 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-systemd-units\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.191815 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190932 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-systemd-units\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.191815 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190947 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-host-run-netns\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.191815 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190950 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1d7b0d2-1555-4c35-a593-d96915d42afe-host-slash\") pod \"iptables-alerter-vgct2\" (UID: \"a1d7b0d2-1555-4c35-a593-d96915d42afe\") " pod="openshift-network-operator/iptables-alerter-vgct2" Jul 06 20:58:37.191815 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190908 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa-registration-dir\") pod \"aws-ebs-csi-driver-node-ntx72\" (UID: \"6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntx72" Jul 06 20:58:37.191815 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190977 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-etc-modprobe-d\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.192744 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190990 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aec1284f-edc5-472a-9552-2d73b756df48-cni-binary-copy\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.192744 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191005 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-etc-kubernetes\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.192744 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.190980 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-host-run-netns\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.192744 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191014 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal" Jul 06 20:58:37.192744 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191049 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-lib-modules\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.192744 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191081 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgv57\" (UniqueName: \"kubernetes.io/projected/6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa-kube-api-access-xgv57\") pod \"aws-ebs-csi-driver-node-ntx72\" (UID: \"6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntx72" Jul 06 20:58:37.192744 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191109 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-host-var-lib-cni-bin\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.192744 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191139 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bb28f09-af91-446b-9f07-6a9a331b2bf5-metrics-certs\") pod \"network-metrics-daemon-qc5r5\" (UID: \"6bb28f09-af91-446b-9f07-6a9a331b2bf5\") " pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:58:37.192744 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191163 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a1d7b0d2-1555-4c35-a593-d96915d42afe-iptables-alerter-script\") pod \"iptables-alerter-vgct2\" (UID: \"a1d7b0d2-1555-4c35-a593-d96915d42afe\") " pod="openshift-network-operator/iptables-alerter-vgct2" Jul 06 20:58:37.192744 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191182 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-host-var-lib-cni-bin\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.192744 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191189 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-log-socket\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.192744 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191206 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8ca96cb1-d98e-4362-b6ad-00dd336bd23d-konnectivity-ca\") pod \"konnectivity-agent-pvvfl\" (UID: \"8ca96cb1-d98e-4362-b6ad-00dd336bd23d\") " pod="kube-system/konnectivity-agent-pvvfl" Jul 06 20:58:37.192744 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191226 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hchk\" (UniqueName: \"kubernetes.io/projected/006c0aa4-b1d8-4e60-af3d-6f994de5c237-kube-api-access-8hchk\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.192744 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191234 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-log-socket\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.192744 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191265 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9511f0e-6c1f-431c-8419-c1adc71f5520-host\") pod \"node-ca-dmgkn\" (UID: \"a9511f0e-6c1f-431c-8419-c1adc71f5520\") " pod="openshift-image-registry/node-ca-dmgkn" Jul 06 20:58:37.193508 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191291 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fcb1b646-77b2-4499-a4c7-4cfabfe3dcca-cnibin\") pod \"multus-additional-cni-plugins-v7sdw\" (UID: \"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca\") " pod="openshift-multus/multus-additional-cni-plugins-v7sdw" Jul 06 20:58:37.193508 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:37.191314 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:58:37.193508 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191321 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-run-systemd\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.193508 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191353 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.193508 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:37.191398 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bb28f09-af91-446b-9f07-6a9a331b2bf5-metrics-certs podName:6bb28f09-af91-446b-9f07-6a9a331b2bf5 nodeName:}" failed. No retries permitted until 2026-07-06 20:58:37.691357838 +0000 UTC m=+2.009581102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bb28f09-af91-446b-9f07-6a9a331b2bf5-metrics-certs") pod "network-metrics-daemon-qc5r5" (UID: "6bb28f09-af91-446b-9f07-6a9a331b2bf5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:58:37.193508 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191402 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.193508 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191403 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-run-systemd\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.193508 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191428 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/64b45669-ed56-41b9-a3ed-1fe5c13508b6-ovn-node-metrics-cert\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.193508 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191454 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/64b45669-ed56-41b9-a3ed-1fe5c13508b6-env-overrides\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.193508 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191477 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0f9a2ff-6f72-4374-a459-b4da598d3533-original-pull-secret\") pod \"global-pull-secret-syncer-dbnt9\" (UID: \"b0f9a2ff-6f72-4374-a459-b4da598d3533\") " pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:58:37.193508 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191515 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aec1284f-edc5-472a-9552-2d73b756df48-multus-daemon-config\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.193508 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191537 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-etc-kubernetes\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.193508 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191562 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj6kq\" (UniqueName: \"kubernetes.io/projected/b3810ec4-264c-4e94-ba9e-ab262e657770-kube-api-access-lj6kq\") pod \"network-check-target-n8x9c\" (UID: \"b3810ec4-264c-4e94-ba9e-ab262e657770\") " pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:58:37.193508 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:37.191600 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:58:37.194236 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191612 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-var-lib-openvswitch\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.194236 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191617 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-etc-kubernetes\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.194236 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191641 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdrvj\" (UniqueName: \"kubernetes.io/projected/fcb1b646-77b2-4499-a4c7-4cfabfe3dcca-kube-api-access-tdrvj\") pod \"multus-additional-cni-plugins-v7sdw\" (UID: \"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca\") " pod="openshift-multus/multus-additional-cni-plugins-v7sdw" Jul 06 20:58:37.194236 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:37.191660 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0f9a2ff-6f72-4374-a459-b4da598d3533-original-pull-secret podName:b0f9a2ff-6f72-4374-a459-b4da598d3533 nodeName:}" failed. No retries permitted until 2026-07-06 20:58:37.691643305 +0000 UTC m=+2.009866564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0f9a2ff-6f72-4374-a459-b4da598d3533-original-pull-secret") pod "global-pull-secret-syncer-dbnt9" (UID: "b0f9a2ff-6f72-4374-a459-b4da598d3533") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:58:37.194236 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191698 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-var-lib-openvswitch\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.194236 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191705 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa-etc-selinux\") pod \"aws-ebs-csi-driver-node-ntx72\" (UID: \"6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntx72" Jul 06 20:58:37.194236 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191735 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-hostroot\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.194236 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191737 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a1d7b0d2-1555-4c35-a593-d96915d42afe-iptables-alerter-script\") pod \"iptables-alerter-vgct2\" (UID: \"a1d7b0d2-1555-4c35-a593-d96915d42afe\") " pod="openshift-network-operator/iptables-alerter-vgct2" Jul 06 20:58:37.194236 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191763 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxbrg\" (UniqueName: \"kubernetes.io/projected/6bb28f09-af91-446b-9f07-6a9a331b2bf5-kube-api-access-hxbrg\") pod \"network-metrics-daemon-qc5r5\" (UID: \"6bb28f09-af91-446b-9f07-6a9a331b2bf5\") " pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:58:37.194236 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191793 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-host-run-ovn-kubernetes\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.194236 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191808 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa-etc-selinux\") pod \"aws-ebs-csi-driver-node-ntx72\" (UID: \"6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntx72" Jul 06 20:58:37.194236 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191815 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aec1284f-edc5-472a-9552-2d73b756df48-hostroot\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.194236 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191818 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/64b45669-ed56-41b9-a3ed-1fe5c13508b6-ovnkube-config\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.194833 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191852 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-sys\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.194833 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191880 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/006c0aa4-b1d8-4e60-af3d-6f994de5c237-etc-tuned\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.194833 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.191958 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64b45669-ed56-41b9-a3ed-1fe5c13508b6-host-run-ovn-kubernetes\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.194833 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.192166 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/64b45669-ed56-41b9-a3ed-1fe5c13508b6-env-overrides\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.194833 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.192380 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aec1284f-edc5-472a-9552-2d73b756df48-multus-daemon-config\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.194833 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.192406 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/64b45669-ed56-41b9-a3ed-1fe5c13508b6-ovnkube-config\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.194833 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.193316 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8ca96cb1-d98e-4362-b6ad-00dd336bd23d-agent-certs\") pod \"konnectivity-agent-pvvfl\" (UID: \"8ca96cb1-d98e-4362-b6ad-00dd336bd23d\") " pod="kube-system/konnectivity-agent-pvvfl" Jul 06 20:58:37.194833 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.194551 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/64b45669-ed56-41b9-a3ed-1fe5c13508b6-ovn-node-metrics-cert\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.195788 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.195524 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-207.ec2.internal" Jul 06 20:58:37.209191 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:37.209155 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:58:37.209191 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:37.209186 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:58:37.209191 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:37.209199 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lj6kq for pod openshift-network-diagnostics/network-check-target-n8x9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:58:37.209525 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:37.209280 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3810ec4-264c-4e94-ba9e-ab262e657770-kube-api-access-lj6kq podName:b3810ec4-264c-4e94-ba9e-ab262e657770 nodeName:}" failed. No retries permitted until 2026-07-06 20:58:37.709261336 +0000 UTC m=+2.027484590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lj6kq" (UniqueName: "kubernetes.io/projected/b3810ec4-264c-4e94-ba9e-ab262e657770-kube-api-access-lj6kq") pod "network-check-target-n8x9c" (UID: "b3810ec4-264c-4e94-ba9e-ab262e657770") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:58:37.210211 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.210176 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxbrg\" (UniqueName: \"kubernetes.io/projected/6bb28f09-af91-446b-9f07-6a9a331b2bf5-kube-api-access-hxbrg\") pod \"network-metrics-daemon-qc5r5\" (UID: \"6bb28f09-af91-446b-9f07-6a9a331b2bf5\") " pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:58:37.211877 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.211847 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kblxn\" (UniqueName: \"kubernetes.io/projected/aec1284f-edc5-472a-9552-2d73b756df48-kube-api-access-kblxn\") pod \"multus-d829k\" (UID: \"aec1284f-edc5-472a-9552-2d73b756df48\") " pod="openshift-multus/multus-d829k" Jul 06 20:58:37.216851 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.216826 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgv57\" (UniqueName: \"kubernetes.io/projected/6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa-kube-api-access-xgv57\") pod \"aws-ebs-csi-driver-node-ntx72\" (UID: \"6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntx72" Jul 06 20:58:37.217464 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.217412 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tnpk\" (UniqueName: \"kubernetes.io/projected/a1d7b0d2-1555-4c35-a593-d96915d42afe-kube-api-access-4tnpk\") pod \"iptables-alerter-vgct2\" (UID: \"a1d7b0d2-1555-4c35-a593-d96915d42afe\") " pod="openshift-network-operator/iptables-alerter-vgct2" Jul 06 20:58:37.228625 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.228587 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmh9x\" (UniqueName: \"kubernetes.io/projected/64b45669-ed56-41b9-a3ed-1fe5c13508b6-kube-api-access-mmh9x\") pod \"ovnkube-node-qjs2n\" (UID: \"64b45669-ed56-41b9-a3ed-1fe5c13508b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.293201 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293163 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-etc-sysctl-conf\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.293201 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293198 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-etc-sysconfig\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.293427 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293220 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/006c0aa4-b1d8-4e60-af3d-6f994de5c237-tmp\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.293427 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293246 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/826779c0-bcd6-4c50-9ec1-0b62c9122657-tmp-dir\") pod \"node-resolver-9mn4b\" (UID: \"826779c0-bcd6-4c50-9ec1-0b62c9122657\") " pod="openshift-dns/node-resolver-9mn4b" Jul 06 20:58:37.293427 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293307 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-etc-sysconfig\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.293427 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293332 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a9511f0e-6c1f-431c-8419-c1adc71f5520-serviceca\") pod \"node-ca-dmgkn\" (UID: \"a9511f0e-6c1f-431c-8419-c1adc71f5520\") " pod="openshift-image-registry/node-ca-dmgkn" Jul 06 20:58:37.293427 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293375 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-etc-sysctl-conf\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.293427 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293376 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fcb1b646-77b2-4499-a4c7-4cfabfe3dcca-system-cni-dir\") pod \"multus-additional-cni-plugins-v7sdw\" (UID: \"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca\") " pod="openshift-multus/multus-additional-cni-plugins-v7sdw" Jul 06 20:58:37.293427 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293423 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fcb1b646-77b2-4499-a4c7-4cfabfe3dcca-system-cni-dir\") pod \"multus-additional-cni-plugins-v7sdw\" (UID: \"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca\") " pod="openshift-multus/multus-additional-cni-plugins-v7sdw" Jul 06 20:58:37.293427 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293427 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-etc-systemd\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.293812 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293461 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-run\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.293812 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293470 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-etc-systemd\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.293812 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293489 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-var-lib-kubelet\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.293812 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293514 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-host\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.293812 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293523 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-run\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.293812 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293549 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fcb1b646-77b2-4499-a4c7-4cfabfe3dcca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v7sdw\" (UID: \"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca\") " pod="openshift-multus/multus-additional-cni-plugins-v7sdw" Jul 06 20:58:37.293812 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293581 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fg74\" (UniqueName: \"kubernetes.io/projected/a9511f0e-6c1f-431c-8419-c1adc71f5520-kube-api-access-7fg74\") pod \"node-ca-dmgkn\" (UID: \"a9511f0e-6c1f-431c-8419-c1adc71f5520\") " pod="openshift-image-registry/node-ca-dmgkn" Jul 06 20:58:37.293812 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293596 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/826779c0-bcd6-4c50-9ec1-0b62c9122657-tmp-dir\") pod \"node-resolver-9mn4b\" (UID: \"826779c0-bcd6-4c50-9ec1-0b62c9122657\") " pod="openshift-dns/node-resolver-9mn4b" Jul 06 20:58:37.293812 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293612 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjchr\" (UniqueName: \"kubernetes.io/projected/826779c0-bcd6-4c50-9ec1-0b62c9122657-kube-api-access-hjchr\") pod \"node-resolver-9mn4b\" (UID: \"826779c0-bcd6-4c50-9ec1-0b62c9122657\") " pod="openshift-dns/node-resolver-9mn4b" Jul 06 20:58:37.293812 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293596 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-var-lib-kubelet\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.293812 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293653 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-host\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.293812 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293650 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-etc-sysctl-d\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.293812 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293724 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fcb1b646-77b2-4499-a4c7-4cfabfe3dcca-cni-binary-copy\") pod \"multus-additional-cni-plugins-v7sdw\" (UID: \"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca\") " pod="openshift-multus/multus-additional-cni-plugins-v7sdw" Jul 06 20:58:37.293812 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293754 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fcb1b646-77b2-4499-a4c7-4cfabfe3dcca-os-release\") pod \"multus-additional-cni-plugins-v7sdw\" (UID: \"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca\") " pod="openshift-multus/multus-additional-cni-plugins-v7sdw" Jul 06 20:58:37.294873 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293779 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fcb1b646-77b2-4499-a4c7-4cfabfe3dcca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v7sdw\" (UID: \"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca\") " pod="openshift-multus/multus-additional-cni-plugins-v7sdw" Jul 06 20:58:37.294873 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293825 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fcb1b646-77b2-4499-a4c7-4cfabfe3dcca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v7sdw\" (UID: \"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca\") " pod="openshift-multus/multus-additional-cni-plugins-v7sdw" Jul 06 20:58:37.294873 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293854 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/826779c0-bcd6-4c50-9ec1-0b62c9122657-hosts-file\") pod \"node-resolver-9mn4b\" (UID: \"826779c0-bcd6-4c50-9ec1-0b62c9122657\") " pod="openshift-dns/node-resolver-9mn4b" Jul 06 20:58:37.294873 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293885 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-etc-modprobe-d\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.294873 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293896 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a9511f0e-6c1f-431c-8419-c1adc71f5520-serviceca\") pod \"node-ca-dmgkn\" (UID: \"a9511f0e-6c1f-431c-8419-c1adc71f5520\") " pod="openshift-image-registry/node-ca-dmgkn" Jul 06 20:58:37.294873 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293923 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-etc-kubernetes\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.294873 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293901 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fcb1b646-77b2-4499-a4c7-4cfabfe3dcca-os-release\") pod \"multus-additional-cni-plugins-v7sdw\" (UID: \"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca\") " pod="openshift-multus/multus-additional-cni-plugins-v7sdw" Jul 06 20:58:37.294873 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293948 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-lib-modules\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.294873 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293991 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/826779c0-bcd6-4c50-9ec1-0b62c9122657-hosts-file\") pod \"node-resolver-9mn4b\" (UID: \"826779c0-bcd6-4c50-9ec1-0b62c9122657\") " pod="openshift-dns/node-resolver-9mn4b" Jul 06 20:58:37.294873 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293995 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-etc-kubernetes\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.294873 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.294014 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hchk\" (UniqueName: \"kubernetes.io/projected/006c0aa4-b1d8-4e60-af3d-6f994de5c237-kube-api-access-8hchk\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.294873 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.293905 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-etc-sysctl-d\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.294873 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.294053 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9511f0e-6c1f-431c-8419-c1adc71f5520-host\") pod \"node-ca-dmgkn\" (UID: \"a9511f0e-6c1f-431c-8419-c1adc71f5520\") " pod="openshift-image-registry/node-ca-dmgkn" Jul 06 20:58:37.294873 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.294082 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fcb1b646-77b2-4499-a4c7-4cfabfe3dcca-cnibin\") pod \"multus-additional-cni-plugins-v7sdw\" (UID: \"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca\") " pod="openshift-multus/multus-additional-cni-plugins-v7sdw" Jul 06 20:58:37.295304 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.294168 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdrvj\" (UniqueName: \"kubernetes.io/projected/fcb1b646-77b2-4499-a4c7-4cfabfe3dcca-kube-api-access-tdrvj\") pod \"multus-additional-cni-plugins-v7sdw\" (UID: \"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca\") " pod="openshift-multus/multus-additional-cni-plugins-v7sdw" Jul 06 20:58:37.295304 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.294206 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-sys\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.295304 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.294210 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-lib-modules\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.295304 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.294212 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fcb1b646-77b2-4499-a4c7-4cfabfe3dcca-cnibin\") pod \"multus-additional-cni-plugins-v7sdw\" (UID: \"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca\") " pod="openshift-multus/multus-additional-cni-plugins-v7sdw" Jul 06 20:58:37.295304 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.294173 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-etc-modprobe-d\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.295304 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.294231 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/006c0aa4-b1d8-4e60-af3d-6f994de5c237-etc-tuned\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.295304 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.294229 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fcb1b646-77b2-4499-a4c7-4cfabfe3dcca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v7sdw\" (UID: \"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca\") " pod="openshift-multus/multus-additional-cni-plugins-v7sdw" Jul 06 20:58:37.295304 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.294248 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9511f0e-6c1f-431c-8419-c1adc71f5520-host\") pod \"node-ca-dmgkn\" (UID: \"a9511f0e-6c1f-431c-8419-c1adc71f5520\") " pod="openshift-image-registry/node-ca-dmgkn" Jul 06 20:58:37.295304 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.294284 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/006c0aa4-b1d8-4e60-af3d-6f994de5c237-sys\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.295304 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.294399 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fcb1b646-77b2-4499-a4c7-4cfabfe3dcca-cni-binary-copy\") pod \"multus-additional-cni-plugins-v7sdw\" (UID: \"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca\") " pod="openshift-multus/multus-additional-cni-plugins-v7sdw" Jul 06 20:58:37.295304 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.294436 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fcb1b646-77b2-4499-a4c7-4cfabfe3dcca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v7sdw\" (UID: \"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca\") " pod="openshift-multus/multus-additional-cni-plugins-v7sdw" Jul 06 20:58:37.295304 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.294627 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fcb1b646-77b2-4499-a4c7-4cfabfe3dcca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v7sdw\" (UID: \"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca\") " pod="openshift-multus/multus-additional-cni-plugins-v7sdw" Jul 06 20:58:37.295874 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.295854 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/006c0aa4-b1d8-4e60-af3d-6f994de5c237-tmp\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.296195 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.296179 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/006c0aa4-b1d8-4e60-af3d-6f994de5c237-etc-tuned\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.306226 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.306201 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdrvj\" (UniqueName: \"kubernetes.io/projected/fcb1b646-77b2-4499-a4c7-4cfabfe3dcca-kube-api-access-tdrvj\") pod \"multus-additional-cni-plugins-v7sdw\" (UID: \"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca\") " pod="openshift-multus/multus-additional-cni-plugins-v7sdw" Jul 06 20:58:37.307343 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.307323 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hchk\" (UniqueName: \"kubernetes.io/projected/006c0aa4-b1d8-4e60-af3d-6f994de5c237-kube-api-access-8hchk\") pod \"tuned-8bxzl\" (UID: \"006c0aa4-b1d8-4e60-af3d-6f994de5c237\") " pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.312429 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.312401 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fg74\" (UniqueName: \"kubernetes.io/projected/a9511f0e-6c1f-431c-8419-c1adc71f5520-kube-api-access-7fg74\") pod \"node-ca-dmgkn\" (UID: \"a9511f0e-6c1f-431c-8419-c1adc71f5520\") " pod="openshift-image-registry/node-ca-dmgkn" Jul 06 20:58:37.324419 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.324385 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjchr\" (UniqueName: \"kubernetes.io/projected/826779c0-bcd6-4c50-9ec1-0b62c9122657-kube-api-access-hjchr\") pod \"node-resolver-9mn4b\" (UID: \"826779c0-bcd6-4c50-9ec1-0b62c9122657\") " pod="openshift-dns/node-resolver-9mn4b" Jul 06 20:58:37.373363 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.373324 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d829k" Jul 06 20:58:37.380651 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.380627 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vgct2" Jul 06 20:58:37.395492 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.395415 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:58:37.418508 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.418474 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pvvfl" Jul 06 20:58:37.441420 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.441380 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntx72" Jul 06 20:58:37.455875 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.455838 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" Jul 06 20:58:37.471084 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.471034 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9mn4b" Jul 06 20:58:37.484647 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.484612 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dmgkn" Jul 06 20:58:37.490865 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.490835 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v7sdw" Jul 06 20:58:37.696991 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.696898 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0f9a2ff-6f72-4374-a459-b4da598d3533-original-pull-secret\") pod \"global-pull-secret-syncer-dbnt9\" (UID: \"b0f9a2ff-6f72-4374-a459-b4da598d3533\") " pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:58:37.697255 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.697026 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bb28f09-af91-446b-9f07-6a9a331b2bf5-metrics-certs\") pod \"network-metrics-daemon-qc5r5\" (UID: \"6bb28f09-af91-446b-9f07-6a9a331b2bf5\") " pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:58:37.697255 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:37.697090 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:58:37.697255 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:37.697153 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:58:37.697255 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:37.697167 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0f9a2ff-6f72-4374-a459-b4da598d3533-original-pull-secret podName:b0f9a2ff-6f72-4374-a459-b4da598d3533 nodeName:}" failed. No retries permitted until 2026-07-06 20:58:38.69714398 +0000 UTC m=+3.015367226 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0f9a2ff-6f72-4374-a459-b4da598d3533-original-pull-secret") pod "global-pull-secret-syncer-dbnt9" (UID: "b0f9a2ff-6f72-4374-a459-b4da598d3533") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:58:37.697255 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:37.697200 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bb28f09-af91-446b-9f07-6a9a331b2bf5-metrics-certs podName:6bb28f09-af91-446b-9f07-6a9a331b2bf5 nodeName:}" failed. No retries permitted until 2026-07-06 20:58:38.69718675 +0000 UTC m=+3.015410004 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bb28f09-af91-446b-9f07-6a9a331b2bf5-metrics-certs") pod "network-metrics-daemon-qc5r5" (UID: "6bb28f09-af91-446b-9f07-6a9a331b2bf5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:58:37.758811 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:37.758760 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod835fe22bba349376f78a29c5926e82d5.slice/crio-1b8c0874fb75adce50186d90d652abb38315950cf2ae5d884993d5ff853ca021 WatchSource:0}: Error finding container 1b8c0874fb75adce50186d90d652abb38315950cf2ae5d884993d5ff853ca021: Status 404 returned error can't find the container with id 1b8c0874fb75adce50186d90d652abb38315950cf2ae5d884993d5ff853ca021 Jul 06 20:58:37.759475 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:37.759439 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64b45669_ed56_41b9_a3ed_1fe5c13508b6.slice/crio-ae113656c39cd6a06f61dfc1cf129534490dacf592c48fa4260e419e4c98d885 WatchSource:0}: Error finding container ae113656c39cd6a06f61dfc1cf129534490dacf592c48fa4260e419e4c98d885: Status 404 returned error can't find the container with id ae113656c39cd6a06f61dfc1cf129534490dacf592c48fa4260e419e4c98d885 Jul 06 20:58:37.760454 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:37.760431 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaec1284f_edc5_472a_9552_2d73b756df48.slice/crio-878c80d4be48b59a729955fb0294d2607ee04a0061e9c2c6845a1a584b30bfcd WatchSource:0}: Error finding container 878c80d4be48b59a729955fb0294d2607ee04a0061e9c2c6845a1a584b30bfcd: Status 404 returned error can't find the container with id 878c80d4be48b59a729955fb0294d2607ee04a0061e9c2c6845a1a584b30bfcd Jul 06 20:58:37.761807 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:37.761783 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a399fb4023b43548b92e7e89de6a41.slice/crio-5aa4cda1897b14bbb5939021b7c4187ff6287ce995a8c56784bce6d59adf12c4 WatchSource:0}: Error finding container 5aa4cda1897b14bbb5939021b7c4187ff6287ce995a8c56784bce6d59adf12c4: Status 404 returned error can't find the container with id 5aa4cda1897b14bbb5939021b7c4187ff6287ce995a8c56784bce6d59adf12c4 Jul 06 20:58:37.770655 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.770632 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 20:58:37.798419 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:37.798375 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj6kq\" (UniqueName: \"kubernetes.io/projected/b3810ec4-264c-4e94-ba9e-ab262e657770-kube-api-access-lj6kq\") pod \"network-check-target-n8x9c\" (UID: \"b3810ec4-264c-4e94-ba9e-ab262e657770\") " pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:58:37.798591 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:37.798533 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:58:37.798591 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:37.798559 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:58:37.798591 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:37.798573 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lj6kq for pod openshift-network-diagnostics/network-check-target-n8x9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:58:37.798704 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:37.798645 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3810ec4-264c-4e94-ba9e-ab262e657770-kube-api-access-lj6kq podName:b3810ec4-264c-4e94-ba9e-ab262e657770 nodeName:}" failed. No retries permitted until 2026-07-06 20:58:38.798622757 +0000 UTC m=+3.116846018 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lj6kq" (UniqueName: "kubernetes.io/projected/b3810ec4-264c-4e94-ba9e-ab262e657770-kube-api-access-lj6kq") pod "network-check-target-n8x9c" (UID: "b3810ec4-264c-4e94-ba9e-ab262e657770") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:58:37.816239 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:37.816196 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod006c0aa4_b1d8_4e60_af3d_6f994de5c237.slice/crio-e411b975a7f01943c3b5e1bfd3451e629230971e31707e1bcb2244533da175eb WatchSource:0}: Error finding container e411b975a7f01943c3b5e1bfd3451e629230971e31707e1bcb2244533da175eb: Status 404 returned error can't find the container with id e411b975a7f01943c3b5e1bfd3451e629230971e31707e1bcb2244533da175eb Jul 06 20:58:37.819575 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:37.819543 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d7b0d2_1555_4c35_a593_d96915d42afe.slice/crio-56c283ee28184929cde74903424af296e96836f625120caf00c44d628119c6c4 WatchSource:0}: Error finding container 56c283ee28184929cde74903424af296e96836f625120caf00c44d628119c6c4: Status 404 returned error can't find the container with id 56c283ee28184929cde74903424af296e96836f625120caf00c44d628119c6c4 Jul 06 20:58:37.820558 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:37.820532 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a64aac5_fb7d_4529_a3d3_28bc33b4e3fa.slice/crio-a9a7c898a6024240733f45e703c20047c0ee730112a7d1f6efcb512f64fef479 WatchSource:0}: Error finding container a9a7c898a6024240733f45e703c20047c0ee730112a7d1f6efcb512f64fef479: Status 404 returned error can't find the container with id a9a7c898a6024240733f45e703c20047c0ee730112a7d1f6efcb512f64fef479 Jul 06 20:58:37.847332 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:37.847298 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9511f0e_6c1f_431c_8419_c1adc71f5520.slice/crio-321e64dc2586a9135110503d468ff4f148c1215c49b67087ab3e0c624f9fef1a WatchSource:0}: Error finding container 321e64dc2586a9135110503d468ff4f148c1215c49b67087ab3e0c624f9fef1a: Status 404 returned error can't find the container with id 321e64dc2586a9135110503d468ff4f148c1215c49b67087ab3e0c624f9fef1a Jul 06 20:58:37.880539 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:37.880493 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb1b646_77b2_4499_a4c7_4cfabfe3dcca.slice/crio-8ab5b2b419d0c00494af78600f8abe92a97e3cfecb8c0bed7a31f07d4a5068d9 WatchSource:0}: Error finding container 8ab5b2b419d0c00494af78600f8abe92a97e3cfecb8c0bed7a31f07d4a5068d9: Status 404 returned error can't find the container with id 8ab5b2b419d0c00494af78600f8abe92a97e3cfecb8c0bed7a31f07d4a5068d9 Jul 06 20:58:37.888029 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:37.887981 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod826779c0_bcd6_4c50_9ec1_0b62c9122657.slice/crio-c128ba189a506a172fe3fecf783ec744a375834e853c202aaeeb8d115a825f41 WatchSource:0}: Error finding container c128ba189a506a172fe3fecf783ec744a375834e853c202aaeeb8d115a825f41: Status 404 returned error can't find the container with id c128ba189a506a172fe3fecf783ec744a375834e853c202aaeeb8d115a825f41 Jul 06 20:58:37.909854 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:58:37.909820 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca96cb1_d98e_4362_b6ad_00dd336bd23d.slice/crio-aa0bf09f9dce57e7ace54829463a6c3ea71421ab26881e336ea6abffece68528 WatchSource:0}: Error finding container aa0bf09f9dce57e7ace54829463a6c3ea71421ab26881e336ea6abffece68528: Status 404 returned error can't find the container with id aa0bf09f9dce57e7ace54829463a6c3ea71421ab26881e336ea6abffece68528 Jul 06 20:58:37.911791 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:37.911763 2577 kuberuntime_manager.go:1459] "Unhandled Error" err="container konnectivity-agent start failed in pod konnectivity-agent-pvvfl_kube-system(8ca96cb1-d98e-4362-b6ad-00dd336bd23d): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Jul 06 20:58:37.911855 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:37.911809 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"konnectivity-agent\" with ErrImagePull: \"pull QPS exceeded\"" pod="kube-system/konnectivity-agent-pvvfl" podUID="8ca96cb1-d98e-4362-b6ad-00dd336bd23d" Jul 06 20:58:38.002040 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:38.001933 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:58:38.095365 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:38.095305 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 20:53:37 +0000 UTC" deadline="2027-12-13 05:53:34.379714208 +0000 UTC" Jul 06 20:58:38.095365 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:38.095358 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12584h54m56.284359335s" Jul 06 20:58:38.167118 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:38.167042 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pvvfl" event={"ID":"8ca96cb1-d98e-4362-b6ad-00dd336bd23d","Type":"ContainerStarted","Data":"aa0bf09f9dce57e7ace54829463a6c3ea71421ab26881e336ea6abffece68528"} Jul 06 20:58:38.168674 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:38.168619 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"konnectivity-agent\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b2e954d9267f62174ff45a44976fa7a47e58d87fef14ffbb9fc0f8184e722280\\\": ErrImagePull: pull QPS exceeded\"" pod="kube-system/konnectivity-agent-pvvfl" podUID="8ca96cb1-d98e-4362-b6ad-00dd336bd23d" Jul 06 20:58:38.168997 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:38.168962 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9mn4b" event={"ID":"826779c0-bcd6-4c50-9ec1-0b62c9122657","Type":"ContainerStarted","Data":"c128ba189a506a172fe3fecf783ec744a375834e853c202aaeeb8d115a825f41"} Jul 06 20:58:38.170274 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:38.170243 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7sdw" event={"ID":"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca","Type":"ContainerStarted","Data":"8ab5b2b419d0c00494af78600f8abe92a97e3cfecb8c0bed7a31f07d4a5068d9"} Jul 06 20:58:38.171673 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:38.171651 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dmgkn" event={"ID":"a9511f0e-6c1f-431c-8419-c1adc71f5520","Type":"ContainerStarted","Data":"321e64dc2586a9135110503d468ff4f148c1215c49b67087ab3e0c624f9fef1a"} Jul 06 20:58:38.172858 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:38.172833 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntx72" event={"ID":"6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa","Type":"ContainerStarted","Data":"a9a7c898a6024240733f45e703c20047c0ee730112a7d1f6efcb512f64fef479"} Jul 06 20:58:38.174119 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:38.174089 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" event={"ID":"006c0aa4-b1d8-4e60-af3d-6f994de5c237","Type":"ContainerStarted","Data":"e411b975a7f01943c3b5e1bfd3451e629230971e31707e1bcb2244533da175eb"} Jul 06 20:58:38.175573 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:38.175542 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" event={"ID":"64b45669-ed56-41b9-a3ed-1fe5c13508b6","Type":"ContainerStarted","Data":"ae113656c39cd6a06f61dfc1cf129534490dacf592c48fa4260e419e4c98d885"} Jul 06 20:58:38.176937 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:38.176903 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal" event={"ID":"90a399fb4023b43548b92e7e89de6a41","Type":"ContainerStarted","Data":"5aa4cda1897b14bbb5939021b7c4187ff6287ce995a8c56784bce6d59adf12c4"} Jul 06 20:58:38.178316 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:38.178206 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vgct2" event={"ID":"a1d7b0d2-1555-4c35-a593-d96915d42afe","Type":"ContainerStarted","Data":"56c283ee28184929cde74903424af296e96836f625120caf00c44d628119c6c4"} Jul 06 20:58:38.179920 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:38.179890 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d829k" event={"ID":"aec1284f-edc5-472a-9552-2d73b756df48","Type":"ContainerStarted","Data":"878c80d4be48b59a729955fb0294d2607ee04a0061e9c2c6845a1a584b30bfcd"} Jul 06 20:58:38.181761 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:38.181731 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-207.ec2.internal" event={"ID":"835fe22bba349376f78a29c5926e82d5","Type":"ContainerStarted","Data":"1b8c0874fb75adce50186d90d652abb38315950cf2ae5d884993d5ff853ca021"} Jul 06 20:58:38.581596 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:38.581559 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 20:58:38.706735 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:38.706689 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bb28f09-af91-446b-9f07-6a9a331b2bf5-metrics-certs\") pod \"network-metrics-daemon-qc5r5\" (UID: \"6bb28f09-af91-446b-9f07-6a9a331b2bf5\") " pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:58:38.706942 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:38.706760 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0f9a2ff-6f72-4374-a459-b4da598d3533-original-pull-secret\") pod \"global-pull-secret-syncer-dbnt9\" (UID: \"b0f9a2ff-6f72-4374-a459-b4da598d3533\") " pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:58:38.707019 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:38.706942 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:58:38.707019 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:38.707008 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0f9a2ff-6f72-4374-a459-b4da598d3533-original-pull-secret podName:b0f9a2ff-6f72-4374-a459-b4da598d3533 nodeName:}" failed. No retries permitted until 2026-07-06 20:58:40.706988552 +0000 UTC m=+5.025211810 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0f9a2ff-6f72-4374-a459-b4da598d3533-original-pull-secret") pod "global-pull-secret-syncer-dbnt9" (UID: "b0f9a2ff-6f72-4374-a459-b4da598d3533") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:58:38.707501 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:38.707475 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:58:38.707610 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:38.707543 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bb28f09-af91-446b-9f07-6a9a331b2bf5-metrics-certs podName:6bb28f09-af91-446b-9f07-6a9a331b2bf5 nodeName:}" failed. No retries permitted until 2026-07-06 20:58:40.707526025 +0000 UTC m=+5.025749270 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bb28f09-af91-446b-9f07-6a9a331b2bf5-metrics-certs") pod "network-metrics-daemon-qc5r5" (UID: "6bb28f09-af91-446b-9f07-6a9a331b2bf5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:58:38.807660 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:38.807611 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj6kq\" (UniqueName: \"kubernetes.io/projected/b3810ec4-264c-4e94-ba9e-ab262e657770-kube-api-access-lj6kq\") pod \"network-check-target-n8x9c\" (UID: \"b3810ec4-264c-4e94-ba9e-ab262e657770\") " pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:58:38.807862 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:38.807831 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:58:38.807862 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:38.807857 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:58:38.807987 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:38.807871 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lj6kq for pod openshift-network-diagnostics/network-check-target-n8x9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:58:38.807987 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:38.807939 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3810ec4-264c-4e94-ba9e-ab262e657770-kube-api-access-lj6kq podName:b3810ec4-264c-4e94-ba9e-ab262e657770 nodeName:}" failed. No retries permitted until 2026-07-06 20:58:40.807917137 +0000 UTC m=+5.126140395 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lj6kq" (UniqueName: "kubernetes.io/projected/b3810ec4-264c-4e94-ba9e-ab262e657770-kube-api-access-lj6kq") pod "network-check-target-n8x9c" (UID: "b3810ec4-264c-4e94-ba9e-ab262e657770") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:58:39.095946 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:39.095833 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 20:53:37 +0000 UTC" deadline="2028-03-11 18:54:43.429161217 +0000 UTC" Jul 06 20:58:39.095946 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:39.095879 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14733h56m4.333287339s" Jul 06 20:58:39.163466 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:39.162409 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:58:39.163466 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:39.162559 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbnt9" podUID="b0f9a2ff-6f72-4374-a459-b4da598d3533" Jul 06 20:58:39.163466 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:39.162890 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:58:39.163466 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:39.163005 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc5r5" podUID="6bb28f09-af91-446b-9f07-6a9a331b2bf5" Jul 06 20:58:39.163466 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:39.163092 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:58:39.163466 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:39.163167 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8x9c" podUID="b3810ec4-264c-4e94-ba9e-ab262e657770" Jul 06 20:58:40.729570 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:40.728565 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bb28f09-af91-446b-9f07-6a9a331b2bf5-metrics-certs\") pod \"network-metrics-daemon-qc5r5\" (UID: \"6bb28f09-af91-446b-9f07-6a9a331b2bf5\") " pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:58:40.729570 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:40.728630 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0f9a2ff-6f72-4374-a459-b4da598d3533-original-pull-secret\") pod \"global-pull-secret-syncer-dbnt9\" (UID: \"b0f9a2ff-6f72-4374-a459-b4da598d3533\") " pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:58:40.729570 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:40.728813 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:58:40.729570 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:40.728889 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0f9a2ff-6f72-4374-a459-b4da598d3533-original-pull-secret podName:b0f9a2ff-6f72-4374-a459-b4da598d3533 nodeName:}" failed. No retries permitted until 2026-07-06 20:58:44.72886624 +0000 UTC m=+9.047089507 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0f9a2ff-6f72-4374-a459-b4da598d3533-original-pull-secret") pod "global-pull-secret-syncer-dbnt9" (UID: "b0f9a2ff-6f72-4374-a459-b4da598d3533") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:58:40.729570 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:40.729446 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:58:40.729570 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:40.729507 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bb28f09-af91-446b-9f07-6a9a331b2bf5-metrics-certs podName:6bb28f09-af91-446b-9f07-6a9a331b2bf5 nodeName:}" failed. No retries permitted until 2026-07-06 20:58:44.729489228 +0000 UTC m=+9.047712486 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bb28f09-af91-446b-9f07-6a9a331b2bf5-metrics-certs") pod "network-metrics-daemon-qc5r5" (UID: "6bb28f09-af91-446b-9f07-6a9a331b2bf5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:58:40.829852 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:40.829806 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj6kq\" (UniqueName: \"kubernetes.io/projected/b3810ec4-264c-4e94-ba9e-ab262e657770-kube-api-access-lj6kq\") pod \"network-check-target-n8x9c\" (UID: \"b3810ec4-264c-4e94-ba9e-ab262e657770\") " pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:58:40.830065 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:40.829997 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:58:40.830065 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:40.830034 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:58:40.830223 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:40.830157 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lj6kq for pod openshift-network-diagnostics/network-check-target-n8x9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:58:40.830288 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:40.830228 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3810ec4-264c-4e94-ba9e-ab262e657770-kube-api-access-lj6kq podName:b3810ec4-264c-4e94-ba9e-ab262e657770 nodeName:}" failed. No retries permitted until 2026-07-06 20:58:44.830206567 +0000 UTC m=+9.148429825 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lj6kq" (UniqueName: "kubernetes.io/projected/b3810ec4-264c-4e94-ba9e-ab262e657770-kube-api-access-lj6kq") pod "network-check-target-n8x9c" (UID: "b3810ec4-264c-4e94-ba9e-ab262e657770") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:58:41.163380 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:41.162303 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:58:41.163380 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:41.162453 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbnt9" podUID="b0f9a2ff-6f72-4374-a459-b4da598d3533" Jul 06 20:58:41.163380 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:41.162936 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:58:41.163380 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:41.163051 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc5r5" podUID="6bb28f09-af91-446b-9f07-6a9a331b2bf5" Jul 06 20:58:41.163380 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:41.163119 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:58:41.163380 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:41.163194 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8x9c" podUID="b3810ec4-264c-4e94-ba9e-ab262e657770" Jul 06 20:58:43.163494 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:43.163350 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:58:43.164275 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:43.163502 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbnt9" podUID="b0f9a2ff-6f72-4374-a459-b4da598d3533" Jul 06 20:58:43.164275 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:43.163966 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:58:43.164275 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:43.164093 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc5r5" podUID="6bb28f09-af91-446b-9f07-6a9a331b2bf5" Jul 06 20:58:43.164275 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:43.164169 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:58:43.164275 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:43.164244 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8x9c" podUID="b3810ec4-264c-4e94-ba9e-ab262e657770" Jul 06 20:58:44.764040 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:44.763924 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0f9a2ff-6f72-4374-a459-b4da598d3533-original-pull-secret\") pod \"global-pull-secret-syncer-dbnt9\" (UID: \"b0f9a2ff-6f72-4374-a459-b4da598d3533\") " pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:58:44.764040 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:44.764037 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bb28f09-af91-446b-9f07-6a9a331b2bf5-metrics-certs\") pod \"network-metrics-daemon-qc5r5\" (UID: \"6bb28f09-af91-446b-9f07-6a9a331b2bf5\") " pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:58:44.764551 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:44.764114 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:58:44.764551 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:44.764153 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:58:44.764551 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:44.764207 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0f9a2ff-6f72-4374-a459-b4da598d3533-original-pull-secret podName:b0f9a2ff-6f72-4374-a459-b4da598d3533 nodeName:}" failed. No retries permitted until 2026-07-06 20:58:52.764185757 +0000 UTC m=+17.082409001 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0f9a2ff-6f72-4374-a459-b4da598d3533-original-pull-secret") pod "global-pull-secret-syncer-dbnt9" (UID: "b0f9a2ff-6f72-4374-a459-b4da598d3533") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:58:44.764551 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:44.764228 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bb28f09-af91-446b-9f07-6a9a331b2bf5-metrics-certs podName:6bb28f09-af91-446b-9f07-6a9a331b2bf5 nodeName:}" failed. No retries permitted until 2026-07-06 20:58:52.764218148 +0000 UTC m=+17.082441386 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bb28f09-af91-446b-9f07-6a9a331b2bf5-metrics-certs") pod "network-metrics-daemon-qc5r5" (UID: "6bb28f09-af91-446b-9f07-6a9a331b2bf5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:58:44.865285 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:44.865243 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj6kq\" (UniqueName: \"kubernetes.io/projected/b3810ec4-264c-4e94-ba9e-ab262e657770-kube-api-access-lj6kq\") pod \"network-check-target-n8x9c\" (UID: \"b3810ec4-264c-4e94-ba9e-ab262e657770\") " pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:58:44.865483 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:44.865420 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:58:44.865483 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:44.865449 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:58:44.865483 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:44.865464 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lj6kq for pod openshift-network-diagnostics/network-check-target-n8x9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:58:44.865619 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:44.865533 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3810ec4-264c-4e94-ba9e-ab262e657770-kube-api-access-lj6kq podName:b3810ec4-264c-4e94-ba9e-ab262e657770 nodeName:}" failed. No retries permitted until 2026-07-06 20:58:52.865512317 +0000 UTC m=+17.183735561 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lj6kq" (UniqueName: "kubernetes.io/projected/b3810ec4-264c-4e94-ba9e-ab262e657770-kube-api-access-lj6kq") pod "network-check-target-n8x9c" (UID: "b3810ec4-264c-4e94-ba9e-ab262e657770") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:58:45.163118 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:45.163035 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:58:45.163118 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:45.163076 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:58:45.163343 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:45.163189 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbnt9" podUID="b0f9a2ff-6f72-4374-a459-b4da598d3533" Jul 06 20:58:45.163343 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:45.163255 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc5r5" podUID="6bb28f09-af91-446b-9f07-6a9a331b2bf5" Jul 06 20:58:45.163343 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:45.163298 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:58:45.163497 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:45.163345 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8x9c" podUID="b3810ec4-264c-4e94-ba9e-ab262e657770" Jul 06 20:58:47.162728 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:47.162594 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:58:47.163209 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:47.162766 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbnt9" podUID="b0f9a2ff-6f72-4374-a459-b4da598d3533" Jul 06 20:58:47.163209 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:47.162798 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:58:47.163209 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:47.162927 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc5r5" podUID="6bb28f09-af91-446b-9f07-6a9a331b2bf5" Jul 06 20:58:47.163209 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:47.162975 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:58:47.163209 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:47.163059 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8x9c" podUID="b3810ec4-264c-4e94-ba9e-ab262e657770" Jul 06 20:58:49.162471 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:49.162434 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:58:49.162943 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:49.162570 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:58:49.162943 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:49.162575 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8x9c" podUID="b3810ec4-264c-4e94-ba9e-ab262e657770" Jul 06 20:58:49.162943 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:49.162577 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:58:49.162943 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:49.162692 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbnt9" podUID="b0f9a2ff-6f72-4374-a459-b4da598d3533" Jul 06 20:58:49.162943 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:49.162775 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc5r5" podUID="6bb28f09-af91-446b-9f07-6a9a331b2bf5" Jul 06 20:58:51.162446 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:51.162410 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:58:51.162929 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:51.162457 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:58:51.162929 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:51.162457 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:58:51.162929 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:51.162534 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbnt9" podUID="b0f9a2ff-6f72-4374-a459-b4da598d3533" Jul 06 20:58:51.162929 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:51.162665 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc5r5" podUID="6bb28f09-af91-446b-9f07-6a9a331b2bf5" Jul 06 20:58:51.162929 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:51.162751 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8x9c" podUID="b3810ec4-264c-4e94-ba9e-ab262e657770" Jul 06 20:58:52.828505 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:52.828464 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bb28f09-af91-446b-9f07-6a9a331b2bf5-metrics-certs\") pod \"network-metrics-daemon-qc5r5\" (UID: \"6bb28f09-af91-446b-9f07-6a9a331b2bf5\") " pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:58:52.828505 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:52.828516 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0f9a2ff-6f72-4374-a459-b4da598d3533-original-pull-secret\") pod \"global-pull-secret-syncer-dbnt9\" (UID: \"b0f9a2ff-6f72-4374-a459-b4da598d3533\") " pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:58:52.828965 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:52.828630 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:58:52.828965 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:52.828633 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:58:52.828965 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:52.828696 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0f9a2ff-6f72-4374-a459-b4da598d3533-original-pull-secret podName:b0f9a2ff-6f72-4374-a459-b4da598d3533 nodeName:}" failed. No retries permitted until 2026-07-06 20:59:08.828680262 +0000 UTC m=+33.146903521 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0f9a2ff-6f72-4374-a459-b4da598d3533-original-pull-secret") pod "global-pull-secret-syncer-dbnt9" (UID: "b0f9a2ff-6f72-4374-a459-b4da598d3533") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:58:52.828965 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:52.828715 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bb28f09-af91-446b-9f07-6a9a331b2bf5-metrics-certs podName:6bb28f09-af91-446b-9f07-6a9a331b2bf5 nodeName:}" failed. No retries permitted until 2026-07-06 20:59:08.828707701 +0000 UTC m=+33.146930938 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bb28f09-af91-446b-9f07-6a9a331b2bf5-metrics-certs") pod "network-metrics-daemon-qc5r5" (UID: "6bb28f09-af91-446b-9f07-6a9a331b2bf5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:58:52.929654 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:52.929611 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj6kq\" (UniqueName: \"kubernetes.io/projected/b3810ec4-264c-4e94-ba9e-ab262e657770-kube-api-access-lj6kq\") pod \"network-check-target-n8x9c\" (UID: \"b3810ec4-264c-4e94-ba9e-ab262e657770\") " pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:58:52.929836 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:52.929785 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:58:52.929836 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:52.929803 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:58:52.929836 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:52.929814 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lj6kq for pod openshift-network-diagnostics/network-check-target-n8x9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:58:52.929945 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:52.929878 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3810ec4-264c-4e94-ba9e-ab262e657770-kube-api-access-lj6kq podName:b3810ec4-264c-4e94-ba9e-ab262e657770 nodeName:}" failed. No retries permitted until 2026-07-06 20:59:08.929864858 +0000 UTC m=+33.248088115 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lj6kq" (UniqueName: "kubernetes.io/projected/b3810ec4-264c-4e94-ba9e-ab262e657770-kube-api-access-lj6kq") pod "network-check-target-n8x9c" (UID: "b3810ec4-264c-4e94-ba9e-ab262e657770") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:58:53.162953 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:53.162854 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:58:53.162953 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:53.162872 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:58:53.163187 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:53.163092 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc5r5" podUID="6bb28f09-af91-446b-9f07-6a9a331b2bf5" Jul 06 20:58:53.163187 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:53.163168 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:58:53.163297 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:53.163270 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbnt9" podUID="b0f9a2ff-6f72-4374-a459-b4da598d3533" Jul 06 20:58:53.163381 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:53.163359 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8x9c" podUID="b3810ec4-264c-4e94-ba9e-ab262e657770" Jul 06 20:58:55.162584 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:55.162540 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:58:55.163100 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:55.162551 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:58:55.163100 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:55.162673 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbnt9" podUID="b0f9a2ff-6f72-4374-a459-b4da598d3533" Jul 06 20:58:55.163100 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:55.162688 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:58:55.163100 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:55.162769 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8x9c" podUID="b3810ec4-264c-4e94-ba9e-ab262e657770" Jul 06 20:58:55.163100 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:55.162853 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc5r5" podUID="6bb28f09-af91-446b-9f07-6a9a331b2bf5" Jul 06 20:58:57.162776 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:57.162728 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:58:57.163424 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:57.162781 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:58:57.163424 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:57.162734 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:58:57.163424 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:57.162885 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc5r5" podUID="6bb28f09-af91-446b-9f07-6a9a331b2bf5" Jul 06 20:58:57.163424 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:57.163035 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8x9c" podUID="b3810ec4-264c-4e94-ba9e-ab262e657770" Jul 06 20:58:57.163424 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:57.163119 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbnt9" podUID="b0f9a2ff-6f72-4374-a459-b4da598d3533" Jul 06 20:58:59.163407 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:59.162973 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:58:59.164254 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:59.163537 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc5r5" podUID="6bb28f09-af91-446b-9f07-6a9a331b2bf5" Jul 06 20:58:59.164254 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:59.163003 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:58:59.164254 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:59.162994 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:58:59.164254 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:59.163638 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbnt9" podUID="b0f9a2ff-6f72-4374-a459-b4da598d3533" Jul 06 20:58:59.164254 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:58:59.163705 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8x9c" podUID="b3810ec4-264c-4e94-ba9e-ab262e657770" Jul 06 20:58:59.236332 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:59.236302 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-207.ec2.internal" event={"ID":"835fe22bba349376f78a29c5926e82d5","Type":"ContainerStarted","Data":"c2c16a35fb263bbd6773e8b17c379e5be839aae9226bb8d7c77ebb99252a9820"} Jul 06 20:58:59.237724 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:59.237701 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" event={"ID":"006c0aa4-b1d8-4e60-af3d-6f994de5c237","Type":"ContainerStarted","Data":"a86a7ed9e1280d190d36bf445e82218faeae9ba37aabbe020498e48852043506"} Jul 06 20:58:59.240494 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:59.240472 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qjs2n_64b45669-ed56-41b9-a3ed-1fe5c13508b6/ovn-acl-logging/0.log" Jul 06 20:58:59.240999 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:59.240968 2577 generic.go:356] "Generic (PLEG): container finished" podID="64b45669-ed56-41b9-a3ed-1fe5c13508b6" containerID="ff0efd6aa63e5b34598441300e530613aa12550931c2663744bd28f70315f707" exitCode=1 Jul 06 20:58:59.241126 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:59.241021 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" event={"ID":"64b45669-ed56-41b9-a3ed-1fe5c13508b6","Type":"ContainerStarted","Data":"05092e6a908dfa5e54de76e99f16e1c60423240a51f9aed6295505906d16f091"} Jul 06 20:58:59.241126 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:59.241086 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" event={"ID":"64b45669-ed56-41b9-a3ed-1fe5c13508b6","Type":"ContainerStarted","Data":"db2cf6998c5ef4d1dd3a6a6d06c71d25e8378dc545c252fc44997706aa75e2e2"} Jul 06 20:58:59.241232 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:59.241135 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" event={"ID":"64b45669-ed56-41b9-a3ed-1fe5c13508b6","Type":"ContainerStarted","Data":"4cf27016c64bc109d00c89abbc7f528d229589536aaf22db3e513fb61bd64409"} Jul 06 20:58:59.241232 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:59.241183 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" event={"ID":"64b45669-ed56-41b9-a3ed-1fe5c13508b6","Type":"ContainerDied","Data":"ff0efd6aa63e5b34598441300e530613aa12550931c2663744bd28f70315f707"} Jul 06 20:58:59.241326 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:59.241232 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" event={"ID":"64b45669-ed56-41b9-a3ed-1fe5c13508b6","Type":"ContainerStarted","Data":"0c7cea404914555f18ac1499b0129468661516cd6b8ef49c3ac00e03db19ecab"} Jul 06 20:58:59.242590 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:59.242565 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d829k" event={"ID":"aec1284f-edc5-472a-9552-2d73b756df48","Type":"ContainerStarted","Data":"20f5af4db02d49e9246f1a207f4a378e3103284a00c8ea4b417e62fb758a2031"} Jul 06 20:58:59.288725 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:59.288680 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-207.ec2.internal" podStartSLOduration=22.288662473 podStartE2EDuration="22.288662473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:58:59.256302393 +0000 UTC m=+23.574525667" watchObservedRunningTime="2026-07-06 20:58:59.288662473 +0000 UTC m=+23.606885732" Jul 06 20:58:59.288922 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:59.288895 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8bxzl" podStartSLOduration=3.210406964 podStartE2EDuration="23.288886721s" totalImagesPullingTime="20.078479757s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:58:59.288501613 +0000 UTC m=+23.606724888" watchObservedRunningTime="2026-07-06 20:58:59.288886721 +0000 UTC m=+23.607109982" Jul 06 20:58:59.312803 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:58:59.312529 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-d829k" podStartSLOduration=2.633653723 podStartE2EDuration="23.312506651s" totalImagesPullingTime="20.678852928s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:58:59.311931953 +0000 UTC m=+23.630155214" watchObservedRunningTime="2026-07-06 20:58:59.312506651 +0000 UTC m=+23.630729912" Jul 06 20:59:00.246804 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:00.246501 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pvvfl" event={"ID":"8ca96cb1-d98e-4362-b6ad-00dd336bd23d","Type":"ContainerStarted","Data":"ab99fcbe37ecdb9979c46b6c6aff44bb6e62a64509c71bc93f76747e28e79342"} Jul 06 20:59:00.248219 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:00.248185 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9mn4b" event={"ID":"826779c0-bcd6-4c50-9ec1-0b62c9122657","Type":"ContainerStarted","Data":"e3e4330f02c8354fde4c7af222b55577868b64bacc448ead4365141dc5ddbd61"} Jul 06 20:59:00.249958 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:00.249930 2577 generic.go:356] "Generic (PLEG): container finished" podID="fcb1b646-77b2-4499-a4c7-4cfabfe3dcca" containerID="c15a33b71f3d3981f39e141088e073c41e5e475052fc02d92a8d0cc643fad309" exitCode=0 Jul 06 20:59:00.250109 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:00.250033 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7sdw" event={"ID":"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca","Type":"ContainerDied","Data":"c15a33b71f3d3981f39e141088e073c41e5e475052fc02d92a8d0cc643fad309"} Jul 06 20:59:00.251919 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:00.251875 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dmgkn" event={"ID":"a9511f0e-6c1f-431c-8419-c1adc71f5520","Type":"ContainerStarted","Data":"80dafccb216cb9379a347bbb0cab1d87b74d4a6582b3c614417e2c687e24bbb8"} Jul 06 20:59:00.253800 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:00.253771 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntx72" event={"ID":"6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa","Type":"ContainerStarted","Data":"cb3aa6ef3d6edf34875edb74a19b903c4aec374bb6e1fcfbf0ba6f6dfacabbc5"} Jul 06 20:59:00.257182 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:00.257111 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qjs2n_64b45669-ed56-41b9-a3ed-1fe5c13508b6/ovn-acl-logging/0.log" Jul 06 20:59:00.257991 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:00.257958 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" event={"ID":"64b45669-ed56-41b9-a3ed-1fe5c13508b6","Type":"ContainerStarted","Data":"fdce513abb1472f6e3658c9c5c91ab2fb592a2d63bf1321a854d0fa1b4c34143"} Jul 06 20:59:00.259857 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:00.259811 2577 generic.go:356] "Generic (PLEG): container finished" podID="90a399fb4023b43548b92e7e89de6a41" containerID="696537002c3f49abf0a5874513e5d7d110d69f966699f2c91ce7ee543eadcbf0" exitCode=0 Jul 06 20:59:00.259973 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:00.259896 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal" event={"ID":"90a399fb4023b43548b92e7e89de6a41","Type":"ContainerDied","Data":"696537002c3f49abf0a5874513e5d7d110d69f966699f2c91ce7ee543eadcbf0"} Jul 06 20:59:00.261750 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:00.261708 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vgct2" event={"ID":"a1d7b0d2-1555-4c35-a593-d96915d42afe","Type":"ContainerStarted","Data":"ddfa120e0e570acd1988c28fab6f3d38f692db2dd6c0cc45e5288e7843a037a5"} Jul 06 20:59:00.267966 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:00.267901 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pvvfl" podStartSLOduration=3.626234887 podStartE2EDuration="24.267882141s" totalImagesPullingTime="20.641647254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-06 20:59:00.267076792 +0000 UTC m=+24.585300052" watchObservedRunningTime="2026-07-06 20:59:00.267882141 +0000 UTC m=+24.586105402" Jul 06 20:59:00.289587 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:00.289535 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vgct2" podStartSLOduration=4.214077671 podStartE2EDuration="24.289517559s" totalImagesPullingTime="20.075439888s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:59:00.28905768 +0000 UTC m=+24.607280937" watchObservedRunningTime="2026-07-06 20:59:00.289517559 +0000 UTC m=+24.607740821" Jul 06 20:59:00.341038 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:00.340967 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dmgkn" podStartSLOduration=4.299384076 podStartE2EDuration="24.340944824s" totalImagesPullingTime="20.041560748s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:59:00.310415372 +0000 UTC m=+24.628638646" watchObservedRunningTime="2026-07-06 20:59:00.340944824 +0000 UTC m=+24.659168085" Jul 06 20:59:00.719479 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:00.719433 2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 20:59:01.128319 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:01.128216 2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T20:59:00.719480632Z","UUID":"f5a979e8-bd37-415b-a92a-09b3fa074adc","Handler":null,"Name":"","Endpoint":""} Jul 06 20:59:01.129985 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:01.129959 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 20:59:01.130134 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:01.130021 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 20:59:01.163134 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:01.163097 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:59:01.163336 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:01.163115 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:59:01.163336 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:01.163231 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbnt9" podUID="b0f9a2ff-6f72-4374-a459-b4da598d3533" Jul 06 20:59:01.163336 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:01.163321 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8x9c" podUID="b3810ec4-264c-4e94-ba9e-ab262e657770" Jul 06 20:59:01.163500 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:01.163115 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:59:01.163500 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:01.163438 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc5r5" podUID="6bb28f09-af91-446b-9f07-6a9a331b2bf5" Jul 06 20:59:01.266568 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:01.266308 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal" event={"ID":"90a399fb4023b43548b92e7e89de6a41","Type":"ContainerStarted","Data":"3be4e7bef6c35972ff31d58f7be22abeb803b508b4c48a599d8cc65e4c5c09f1"} Jul 06 20:59:01.271634 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:01.271600 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntx72" event={"ID":"6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa","Type":"ContainerStarted","Data":"207a51d8fce90ebeb4a5a50fb0899d92710ba9d53a1ed697c9f28f3e8b00c09a"} Jul 06 20:59:01.285899 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:01.285836 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9mn4b" podStartSLOduration=5.422023781 podStartE2EDuration="25.285811244s" totalImagesPullingTime="19.863787463s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:59:00.389785607 +0000 UTC m=+24.708008867" watchObservedRunningTime="2026-07-06 20:59:01.285811244 +0000 UTC m=+25.604034505" Jul 06 20:59:01.286748 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:01.286683 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-207.ec2.internal" podStartSLOduration=24.286664105 podStartE2EDuration="24.286664105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:59:01.286051329 +0000 UTC m=+25.604274590" watchObservedRunningTime="2026-07-06 20:59:01.286664105 +0000 UTC m=+25.604887369" Jul 06 20:59:02.276417 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:02.276376 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntx72" event={"ID":"6a64aac5-fb7d-4529-a3d3-28bc33b4e3fa","Type":"ContainerStarted","Data":"e5ddc3b8c541aea8c72ac4b5cc7eb6c263414fe171c7dcad12ea44e6a26dc5a5"} Jul 06 20:59:02.280681 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:02.280652 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qjs2n_64b45669-ed56-41b9-a3ed-1fe5c13508b6/ovn-acl-logging/0.log" Jul 06 20:59:02.281111 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:02.281081 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" event={"ID":"64b45669-ed56-41b9-a3ed-1fe5c13508b6","Type":"ContainerStarted","Data":"d244434f513cb9be13c808b8a566d73d5e1210094bc8655e7f35a3db673d22aa"} Jul 06 20:59:03.163027 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:03.162973 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:59:03.163027 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:03.163021 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:59:03.163267 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:03.163041 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:59:03.163267 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:03.163116 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8x9c" podUID="b3810ec4-264c-4e94-ba9e-ab262e657770" Jul 06 20:59:03.163267 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:03.163163 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbnt9" podUID="b0f9a2ff-6f72-4374-a459-b4da598d3533" Jul 06 20:59:03.163267 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:03.163241 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc5r5" podUID="6bb28f09-af91-446b-9f07-6a9a331b2bf5" Jul 06 20:59:04.073060 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:04.072850 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pvvfl" Jul 06 20:59:04.073563 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:04.073455 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pvvfl" Jul 06 20:59:04.090168 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:04.090106 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntx72" podStartSLOduration=5.414449357 podStartE2EDuration="28.090088012s" totalImagesPullingTime="22.675638655s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:58:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:59:02.313824176 +0000 UTC m=+26.632047436" watchObservedRunningTime="2026-07-06 20:59:04.090088012 +0000 UTC m=+28.408311271" Jul 06 20:59:04.292279 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:04.292250 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qjs2n_64b45669-ed56-41b9-a3ed-1fe5c13508b6/ovn-acl-logging/0.log" Jul 06 20:59:04.292658 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:04.292630 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" event={"ID":"64b45669-ed56-41b9-a3ed-1fe5c13508b6","Type":"ContainerStarted","Data":"196fef76d540c2ca927a4419105103ceb8a1ebd324c9bf85c69555d97534b014"} Jul 06 20:59:04.292844 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:04.292814 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pvvfl" Jul 06 20:59:04.293037 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:04.292997 2577 scope.go:117] "RemoveContainer" containerID="ff0efd6aa63e5b34598441300e530613aa12550931c2663744bd28f70315f707" Jul 06 20:59:04.293322 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:04.293305 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pvvfl" Jul 06 20:59:05.163375 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:05.163252 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:59:05.163375 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:05.163287 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:59:05.163375 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:05.163376 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:59:05.164261 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:05.163396 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbnt9" podUID="b0f9a2ff-6f72-4374-a459-b4da598d3533" Jul 06 20:59:05.164261 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:05.163493 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8x9c" podUID="b3810ec4-264c-4e94-ba9e-ab262e657770" Jul 06 20:59:05.164261 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:05.163582 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc5r5" podUID="6bb28f09-af91-446b-9f07-6a9a331b2bf5" Jul 06 20:59:05.300230 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:05.300199 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qjs2n_64b45669-ed56-41b9-a3ed-1fe5c13508b6/ovn-acl-logging/0.log" Jul 06 20:59:05.300735 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:05.300701 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" event={"ID":"64b45669-ed56-41b9-a3ed-1fe5c13508b6","Type":"ContainerStarted","Data":"97ff063601c884fdcf1d8723b6a77dc3552416c31f48e53c70b6a916067217f1"} Jul 06 20:59:05.301121 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:05.301045 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:59:05.301232 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:05.301148 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:59:05.301232 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:05.301199 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:59:05.319598 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:05.319560 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:59:05.319762 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:05.319668 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:59:05.336159 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:05.336093 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" podStartSLOduration=8.535618616 podStartE2EDuration="29.336068229s" totalImagesPullingTime="20.800449613s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:59:05.334642265 +0000 UTC m=+29.652865536" watchObservedRunningTime="2026-07-06 20:59:05.336068229 +0000 UTC m=+29.654291486" Jul 06 20:59:05.723922 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:05.723637 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dbnt9"] Jul 06 20:59:05.724104 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:05.724090 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:59:05.724240 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:05.724210 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbnt9" podUID="b0f9a2ff-6f72-4374-a459-b4da598d3533" Jul 06 20:59:05.725955 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:05.725788 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qc5r5"] Jul 06 20:59:05.725955 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:05.725933 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:59:05.726774 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:05.726081 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc5r5" podUID="6bb28f09-af91-446b-9f07-6a9a331b2bf5" Jul 06 20:59:05.726774 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:05.726519 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n8x9c"] Jul 06 20:59:05.726774 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:05.726626 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:59:05.726774 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:05.726755 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8x9c" podUID="b3810ec4-264c-4e94-ba9e-ab262e657770" Jul 06 20:59:07.162860 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:07.162759 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:59:07.162860 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:07.162792 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:59:07.162860 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:07.162759 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:59:07.163531 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:07.162895 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8x9c" podUID="b3810ec4-264c-4e94-ba9e-ab262e657770" Jul 06 20:59:07.163531 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:07.162963 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbnt9" podUID="b0f9a2ff-6f72-4374-a459-b4da598d3533" Jul 06 20:59:07.163531 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:07.163055 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc5r5" podUID="6bb28f09-af91-446b-9f07-6a9a331b2bf5" Jul 06 20:59:07.307042 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:07.306987 2577 generic.go:356] "Generic (PLEG): container finished" podID="fcb1b646-77b2-4499-a4c7-4cfabfe3dcca" containerID="28e413ed0dc87df975e0b7ba813307ddb46b0df8cff098c2df6b1ab887a7fded" exitCode=0 Jul 06 20:59:07.307246 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:07.307079 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7sdw" event={"ID":"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca","Type":"ContainerDied","Data":"28e413ed0dc87df975e0b7ba813307ddb46b0df8cff098c2df6b1ab887a7fded"} Jul 06 20:59:08.849167 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:08.849125 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0f9a2ff-6f72-4374-a459-b4da598d3533-original-pull-secret\") pod \"global-pull-secret-syncer-dbnt9\" (UID: \"b0f9a2ff-6f72-4374-a459-b4da598d3533\") " pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:59:08.849598 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:08.849285 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bb28f09-af91-446b-9f07-6a9a331b2bf5-metrics-certs\") pod \"network-metrics-daemon-qc5r5\" (UID: \"6bb28f09-af91-446b-9f07-6a9a331b2bf5\") " pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:59:08.849598 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:08.849291 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 20:59:08.849598 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:08.849393 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:59:08.849598 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:08.849402 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0f9a2ff-6f72-4374-a459-b4da598d3533-original-pull-secret podName:b0f9a2ff-6f72-4374-a459-b4da598d3533 nodeName:}" failed. No retries permitted until 2026-07-06 20:59:40.849386797 +0000 UTC m=+65.167610034 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0f9a2ff-6f72-4374-a459-b4da598d3533-original-pull-secret") pod "global-pull-secret-syncer-dbnt9" (UID: "b0f9a2ff-6f72-4374-a459-b4da598d3533") : object "kube-system"/"original-pull-secret" not registered Jul 06 20:59:08.849598 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:08.849435 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bb28f09-af91-446b-9f07-6a9a331b2bf5-metrics-certs podName:6bb28f09-af91-446b-9f07-6a9a331b2bf5 nodeName:}" failed. No retries permitted until 2026-07-06 20:59:40.849422072 +0000 UTC m=+65.167645310 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6bb28f09-af91-446b-9f07-6a9a331b2bf5-metrics-certs") pod "network-metrics-daemon-qc5r5" (UID: "6bb28f09-af91-446b-9f07-6a9a331b2bf5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 20:59:08.950503 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:08.950461 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj6kq\" (UniqueName: \"kubernetes.io/projected/b3810ec4-264c-4e94-ba9e-ab262e657770-kube-api-access-lj6kq\") pod \"network-check-target-n8x9c\" (UID: \"b3810ec4-264c-4e94-ba9e-ab262e657770\") " pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:59:08.950655 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:08.950622 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 20:59:08.950655 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:08.950645 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 20:59:08.950655 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:08.950656 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lj6kq for pod openshift-network-diagnostics/network-check-target-n8x9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:59:08.950791 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:08.950708 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3810ec4-264c-4e94-ba9e-ab262e657770-kube-api-access-lj6kq podName:b3810ec4-264c-4e94-ba9e-ab262e657770 nodeName:}" failed. No retries permitted until 2026-07-06 20:59:40.950694099 +0000 UTC m=+65.268917337 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lj6kq" (UniqueName: "kubernetes.io/projected/b3810ec4-264c-4e94-ba9e-ab262e657770-kube-api-access-lj6kq") pod "network-check-target-n8x9c" (UID: "b3810ec4-264c-4e94-ba9e-ab262e657770") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 20:59:09.162809 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:09.162773 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:59:09.162979 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:09.162817 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:59:09.162979 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:09.162819 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:59:09.162979 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:09.162877 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8x9c" podUID="b3810ec4-264c-4e94-ba9e-ab262e657770" Jul 06 20:59:09.162979 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:09.162970 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc5r5" podUID="6bb28f09-af91-446b-9f07-6a9a331b2bf5" Jul 06 20:59:09.163116 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:09.163052 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbnt9" podUID="b0f9a2ff-6f72-4374-a459-b4da598d3533" Jul 06 20:59:09.314524 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:09.314487 2577 generic.go:356] "Generic (PLEG): container finished" podID="fcb1b646-77b2-4499-a4c7-4cfabfe3dcca" containerID="2e81a26528d3ce7a89f5f94c8c02b21bf53583634a1b4bdb4f4a33b0c2372ef4" exitCode=0 Jul 06 20:59:09.314670 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:09.314548 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7sdw" event={"ID":"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca","Type":"ContainerDied","Data":"2e81a26528d3ce7a89f5f94c8c02b21bf53583634a1b4bdb4f4a33b0c2372ef4"} Jul 06 20:59:11.162453 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.162248 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:59:11.162809 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.162247 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:59:11.162809 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:11.162544 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8x9c" podUID="b3810ec4-264c-4e94-ba9e-ab262e657770" Jul 06 20:59:11.162809 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.162248 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:59:11.162809 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:11.162627 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbnt9" podUID="b0f9a2ff-6f72-4374-a459-b4da598d3533" Jul 06 20:59:11.162809 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:11.162707 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qc5r5" podUID="6bb28f09-af91-446b-9f07-6a9a331b2bf5" Jul 06 20:59:11.325194 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.325093 2577 generic.go:356] "Generic (PLEG): container finished" podID="fcb1b646-77b2-4499-a4c7-4cfabfe3dcca" containerID="5c66ef8403107415b3be762c43b1b0c9dd65753f2c3467a8438ce950c9b101e2" exitCode=0 Jul 06 20:59:11.325194 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.325132 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7sdw" event={"ID":"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca","Type":"ContainerDied","Data":"5c66ef8403107415b3be762c43b1b0c9dd65753f2c3467a8438ce950c9b101e2"} Jul 06 20:59:11.459206 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.459177 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-207.ec2.internal" event="NodeReady" Jul 06 20:59:11.459375 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.459329 2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 20:59:11.503759 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.503723 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-645d5b8597-84426"] Jul 06 20:59:11.527348 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.527318 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xj2d6"] Jul 06 20:59:11.527492 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.527439 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:11.530686 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.530647 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 20:59:11.530686 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.530682 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dhdp7\"" Jul 06 20:59:11.530880 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.530788 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 20:59:11.530880 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.530799 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 20:59:11.536925 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.536900 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 20:59:11.543624 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.543594 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xq4l4"] Jul 06 20:59:11.543624 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.543619 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xj2d6" Jul 06 20:59:11.545908 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.545880 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-27mml\"" Jul 06 20:59:11.546284 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.546262 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 20:59:11.546392 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.546292 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 20:59:11.567608 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.567573 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-645d5b8597-84426"] Jul 06 20:59:11.567608 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.567613 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xj2d6"] Jul 06 20:59:11.567841 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.567631 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xq4l4"] Jul 06 20:59:11.567841 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.567662 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-79v7b"] Jul 06 20:59:11.567841 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.567733 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xq4l4" Jul 06 20:59:11.567968 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.567934 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ad03b367-ccbf-4b95-99f4-29e492b386ac-registry-certificates\") pod \"image-registry-645d5b8597-84426\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:11.568038 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.567973 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx9c7\" (UniqueName: \"kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-kube-api-access-sx9c7\") pod \"image-registry-645d5b8597-84426\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:11.568097 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.568026 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8886f6e5-67fc-4fd8-ad7b-95d98e776a1f-config-volume\") pod \"dns-default-xj2d6\" (UID: \"8886f6e5-67fc-4fd8-ad7b-95d98e776a1f\") " pod="openshift-dns/dns-default-xj2d6" Jul 06 20:59:11.568097 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.568068 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ad03b367-ccbf-4b95-99f4-29e492b386ac-ca-trust-extracted\") pod \"image-registry-645d5b8597-84426\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:11.568184 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.568095 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ad03b367-ccbf-4b95-99f4-29e492b386ac-installation-pull-secrets\") pod \"image-registry-645d5b8597-84426\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:11.568184 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.568151 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8886f6e5-67fc-4fd8-ad7b-95d98e776a1f-metrics-tls\") pod \"dns-default-xj2d6\" (UID: \"8886f6e5-67fc-4fd8-ad7b-95d98e776a1f\") " pod="openshift-dns/dns-default-xj2d6" Jul 06 20:59:11.568278 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.568176 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-bound-sa-token\") pod \"image-registry-645d5b8597-84426\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:11.568278 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.568226 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5pvt\" (UniqueName: \"kubernetes.io/projected/8886f6e5-67fc-4fd8-ad7b-95d98e776a1f-kube-api-access-p5pvt\") pod \"dns-default-xj2d6\" (UID: \"8886f6e5-67fc-4fd8-ad7b-95d98e776a1f\") " pod="openshift-dns/dns-default-xj2d6" Jul 06 20:59:11.568278 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.568250 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ad03b367-ccbf-4b95-99f4-29e492b386ac-trusted-ca\") pod \"image-registry-645d5b8597-84426\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:11.568475 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.568447 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ad03b367-ccbf-4b95-99f4-29e492b386ac-image-registry-private-configuration\") pod \"image-registry-645d5b8597-84426\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:11.568588 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.568513 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-registry-tls\") pod \"image-registry-645d5b8597-84426\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:11.568588 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.568554 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8886f6e5-67fc-4fd8-ad7b-95d98e776a1f-tmp-dir\") pod \"dns-default-xj2d6\" (UID: \"8886f6e5-67fc-4fd8-ad7b-95d98e776a1f\") " pod="openshift-dns/dns-default-xj2d6" Jul 06 20:59:11.570834 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.570810 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 20:59:11.570970 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.570862 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 20:59:11.571305 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.571292 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-85slc\"" Jul 06 20:59:11.571369 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.571292 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 20:59:11.571369 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.571314 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 20:59:11.587864 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.587770 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-79v7b"] Jul 06 20:59:11.588029 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.587900 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-79v7b" Jul 06 20:59:11.590486 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.590458 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 20:59:11.590608 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.590573 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jz828\"" Jul 06 20:59:11.590841 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.590823 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 20:59:11.590918 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.590852 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 20:59:11.669436 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.669396 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ad03b367-ccbf-4b95-99f4-29e492b386ac-registry-certificates\") pod \"image-registry-645d5b8597-84426\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:11.669631 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.669442 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx9c7\" (UniqueName: \"kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-kube-api-access-sx9c7\") pod \"image-registry-645d5b8597-84426\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:11.669631 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.669475 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8886f6e5-67fc-4fd8-ad7b-95d98e776a1f-config-volume\") pod \"dns-default-xj2d6\" (UID: \"8886f6e5-67fc-4fd8-ad7b-95d98e776a1f\") " pod="openshift-dns/dns-default-xj2d6" Jul 06 20:59:11.669631 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.669501 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ad03b367-ccbf-4b95-99f4-29e492b386ac-ca-trust-extracted\") pod \"image-registry-645d5b8597-84426\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:11.669631 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.669527 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7013baf3-f119-4c87-9c16-a9972cb124a8-crio-socket\") pod \"insights-runtime-extractor-xq4l4\" (UID: \"7013baf3-f119-4c87-9c16-a9972cb124a8\") " pod="openshift-insights/insights-runtime-extractor-xq4l4" Jul 06 20:59:11.669631 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.669556 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ad03b367-ccbf-4b95-99f4-29e492b386ac-installation-pull-secrets\") pod \"image-registry-645d5b8597-84426\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:11.669631 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.669589 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8886f6e5-67fc-4fd8-ad7b-95d98e776a1f-metrics-tls\") pod \"dns-default-xj2d6\" (UID: \"8886f6e5-67fc-4fd8-ad7b-95d98e776a1f\") " pod="openshift-dns/dns-default-xj2d6" Jul 06 20:59:11.669916 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.669636 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-bound-sa-token\") pod \"image-registry-645d5b8597-84426\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:11.669916 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.669677 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tck2\" (UniqueName: \"kubernetes.io/projected/09d8ee76-d883-42ab-871d-32be26a2f4f5-kube-api-access-2tck2\") pod \"ingress-canary-79v7b\" (UID: \"09d8ee76-d883-42ab-871d-32be26a2f4f5\") " pod="openshift-ingress-canary/ingress-canary-79v7b" Jul 06 20:59:11.669916 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.669778 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5pvt\" (UniqueName: \"kubernetes.io/projected/8886f6e5-67fc-4fd8-ad7b-95d98e776a1f-kube-api-access-p5pvt\") pod \"dns-default-xj2d6\" (UID: \"8886f6e5-67fc-4fd8-ad7b-95d98e776a1f\") " pod="openshift-dns/dns-default-xj2d6" Jul 06 20:59:11.669916 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:11.669883 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:59:11.670096 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.669917 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ad03b367-ccbf-4b95-99f4-29e492b386ac-trusted-ca\") pod \"image-registry-645d5b8597-84426\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:11.670096 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.669980 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ad03b367-ccbf-4b95-99f4-29e492b386ac-image-registry-private-configuration\") pod \"image-registry-645d5b8597-84426\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:11.670096 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.670001 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ad03b367-ccbf-4b95-99f4-29e492b386ac-ca-trust-extracted\") pod \"image-registry-645d5b8597-84426\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:11.670096 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.670060 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-registry-tls\") pod \"image-registry-645d5b8597-84426\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:11.670096 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:11.670086 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8886f6e5-67fc-4fd8-ad7b-95d98e776a1f-metrics-tls podName:8886f6e5-67fc-4fd8-ad7b-95d98e776a1f nodeName:}" failed. No retries permitted until 2026-07-06 20:59:12.170059454 +0000 UTC m=+36.488282697 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8886f6e5-67fc-4fd8-ad7b-95d98e776a1f-metrics-tls") pod "dns-default-xj2d6" (UID: "8886f6e5-67fc-4fd8-ad7b-95d98e776a1f") : secret "dns-default-metrics-tls" not found Jul 06 20:59:11.670310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.670125 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7013baf3-f119-4c87-9c16-a9972cb124a8-data-volume\") pod \"insights-runtime-extractor-xq4l4\" (UID: \"7013baf3-f119-4c87-9c16-a9972cb124a8\") " pod="openshift-insights/insights-runtime-extractor-xq4l4" Jul 06 20:59:11.670310 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:11.670143 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 20:59:11.670310 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:11.670159 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-645d5b8597-84426: secret "image-registry-tls" not found Jul 06 20:59:11.670310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.670161 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlnr8\" (UniqueName: \"kubernetes.io/projected/7013baf3-f119-4c87-9c16-a9972cb124a8-kube-api-access-zlnr8\") pod \"insights-runtime-extractor-xq4l4\" (UID: \"7013baf3-f119-4c87-9c16-a9972cb124a8\") " pod="openshift-insights/insights-runtime-extractor-xq4l4" Jul 06 20:59:11.670310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.670198 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09d8ee76-d883-42ab-871d-32be26a2f4f5-cert\") pod \"ingress-canary-79v7b\" (UID: \"09d8ee76-d883-42ab-871d-32be26a2f4f5\") " pod="openshift-ingress-canary/ingress-canary-79v7b" Jul 06 20:59:11.670310 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:11.670215 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-registry-tls podName:ad03b367-ccbf-4b95-99f4-29e492b386ac nodeName:}" failed. No retries permitted until 2026-07-06 20:59:12.170199039 +0000 UTC m=+36.488422302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-registry-tls") pod "image-registry-645d5b8597-84426" (UID: "ad03b367-ccbf-4b95-99f4-29e492b386ac") : secret "image-registry-tls" not found Jul 06 20:59:11.670310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.670238 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ad03b367-ccbf-4b95-99f4-29e492b386ac-registry-certificates\") pod \"image-registry-645d5b8597-84426\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:11.670310 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.670260 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8886f6e5-67fc-4fd8-ad7b-95d98e776a1f-tmp-dir\") pod \"dns-default-xj2d6\" (UID: \"8886f6e5-67fc-4fd8-ad7b-95d98e776a1f\") " pod="openshift-dns/dns-default-xj2d6" Jul 06 20:59:11.670705 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.670307 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7013baf3-f119-4c87-9c16-a9972cb124a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xq4l4\" (UID: \"7013baf3-f119-4c87-9c16-a9972cb124a8\") " pod="openshift-insights/insights-runtime-extractor-xq4l4" Jul 06 20:59:11.670705 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.670336 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7013baf3-f119-4c87-9c16-a9972cb124a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xq4l4\" (UID: \"7013baf3-f119-4c87-9c16-a9972cb124a8\") " pod="openshift-insights/insights-runtime-extractor-xq4l4" Jul 06 20:59:11.670705 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.670530 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8886f6e5-67fc-4fd8-ad7b-95d98e776a1f-tmp-dir\") pod \"dns-default-xj2d6\" (UID: \"8886f6e5-67fc-4fd8-ad7b-95d98e776a1f\") " pod="openshift-dns/dns-default-xj2d6" Jul 06 20:59:11.670841 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.670818 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8886f6e5-67fc-4fd8-ad7b-95d98e776a1f-config-volume\") pod \"dns-default-xj2d6\" (UID: \"8886f6e5-67fc-4fd8-ad7b-95d98e776a1f\") " pod="openshift-dns/dns-default-xj2d6" Jul 06 20:59:11.670968 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.670948 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ad03b367-ccbf-4b95-99f4-29e492b386ac-trusted-ca\") pod \"image-registry-645d5b8597-84426\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:11.674274 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.674246 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ad03b367-ccbf-4b95-99f4-29e492b386ac-installation-pull-secrets\") pod \"image-registry-645d5b8597-84426\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:11.674450 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.674248 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ad03b367-ccbf-4b95-99f4-29e492b386ac-image-registry-private-configuration\") pod \"image-registry-645d5b8597-84426\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:11.679095 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.679069 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-bound-sa-token\") pod \"image-registry-645d5b8597-84426\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:11.679557 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.679537 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx9c7\" (UniqueName: \"kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-kube-api-access-sx9c7\") pod \"image-registry-645d5b8597-84426\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:11.679902 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.679881 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5pvt\" (UniqueName: \"kubernetes.io/projected/8886f6e5-67fc-4fd8-ad7b-95d98e776a1f-kube-api-access-p5pvt\") pod \"dns-default-xj2d6\" (UID: \"8886f6e5-67fc-4fd8-ad7b-95d98e776a1f\") " pod="openshift-dns/dns-default-xj2d6" Jul 06 20:59:11.771274 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.771236 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7013baf3-f119-4c87-9c16-a9972cb124a8-data-volume\") pod \"insights-runtime-extractor-xq4l4\" (UID: \"7013baf3-f119-4c87-9c16-a9972cb124a8\") " pod="openshift-insights/insights-runtime-extractor-xq4l4" Jul 06 20:59:11.771274 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.771279 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlnr8\" (UniqueName: \"kubernetes.io/projected/7013baf3-f119-4c87-9c16-a9972cb124a8-kube-api-access-zlnr8\") pod \"insights-runtime-extractor-xq4l4\" (UID: \"7013baf3-f119-4c87-9c16-a9972cb124a8\") " pod="openshift-insights/insights-runtime-extractor-xq4l4" Jul 06 20:59:11.771543 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.771302 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09d8ee76-d883-42ab-871d-32be26a2f4f5-cert\") pod \"ingress-canary-79v7b\" (UID: \"09d8ee76-d883-42ab-871d-32be26a2f4f5\") " pod="openshift-ingress-canary/ingress-canary-79v7b" Jul 06 20:59:11.771543 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.771321 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7013baf3-f119-4c87-9c16-a9972cb124a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xq4l4\" (UID: \"7013baf3-f119-4c87-9c16-a9972cb124a8\") " pod="openshift-insights/insights-runtime-extractor-xq4l4" Jul 06 20:59:11.771543 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.771338 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7013baf3-f119-4c87-9c16-a9972cb124a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xq4l4\" (UID: \"7013baf3-f119-4c87-9c16-a9972cb124a8\") " pod="openshift-insights/insights-runtime-extractor-xq4l4" Jul 06 20:59:11.771543 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.771367 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7013baf3-f119-4c87-9c16-a9972cb124a8-crio-socket\") pod \"insights-runtime-extractor-xq4l4\" (UID: \"7013baf3-f119-4c87-9c16-a9972cb124a8\") " pod="openshift-insights/insights-runtime-extractor-xq4l4" Jul 06 20:59:11.771543 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.771425 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tck2\" (UniqueName: \"kubernetes.io/projected/09d8ee76-d883-42ab-871d-32be26a2f4f5-kube-api-access-2tck2\") pod \"ingress-canary-79v7b\" (UID: \"09d8ee76-d883-42ab-871d-32be26a2f4f5\") " pod="openshift-ingress-canary/ingress-canary-79v7b" Jul 06 20:59:11.771543 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:11.771435 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 20:59:11.771543 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:11.771451 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 20:59:11.771543 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:11.771533 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7013baf3-f119-4c87-9c16-a9972cb124a8-insights-runtime-extractor-tls podName:7013baf3-f119-4c87-9c16-a9972cb124a8 nodeName:}" failed. No retries permitted until 2026-07-06 20:59:12.271509386 +0000 UTC m=+36.589732644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7013baf3-f119-4c87-9c16-a9972cb124a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xq4l4" (UID: "7013baf3-f119-4c87-9c16-a9972cb124a8") : secret "insights-runtime-extractor-tls" not found Jul 06 20:59:11.771867 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:11.771553 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7013baf3-f119-4c87-9c16-a9972cb124a8-kube-rbac-proxy-cm podName:7013baf3-f119-4c87-9c16-a9972cb124a8 nodeName:}" failed. No retries permitted until 2026-07-06 20:59:12.271543535 +0000 UTC m=+36.589766773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7013baf3-f119-4c87-9c16-a9972cb124a8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xq4l4" (UID: "7013baf3-f119-4c87-9c16-a9972cb124a8") : configmap "kube-rbac-proxy" not found Jul 06 20:59:11.771867 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.771656 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7013baf3-f119-4c87-9c16-a9972cb124a8-data-volume\") pod \"insights-runtime-extractor-xq4l4\" (UID: \"7013baf3-f119-4c87-9c16-a9972cb124a8\") " pod="openshift-insights/insights-runtime-extractor-xq4l4" Jul 06 20:59:11.771867 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.771690 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7013baf3-f119-4c87-9c16-a9972cb124a8-crio-socket\") pod \"insights-runtime-extractor-xq4l4\" (UID: \"7013baf3-f119-4c87-9c16-a9972cb124a8\") " pod="openshift-insights/insights-runtime-extractor-xq4l4" Jul 06 20:59:11.773739 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.773721 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09d8ee76-d883-42ab-871d-32be26a2f4f5-cert\") pod \"ingress-canary-79v7b\" (UID: \"09d8ee76-d883-42ab-871d-32be26a2f4f5\") " pod="openshift-ingress-canary/ingress-canary-79v7b" Jul 06 20:59:11.783299 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.783261 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlnr8\" (UniqueName: \"kubernetes.io/projected/7013baf3-f119-4c87-9c16-a9972cb124a8-kube-api-access-zlnr8\") pod \"insights-runtime-extractor-xq4l4\" (UID: \"7013baf3-f119-4c87-9c16-a9972cb124a8\") " pod="openshift-insights/insights-runtime-extractor-xq4l4" Jul 06 20:59:11.783432 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.783334 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tck2\" (UniqueName: \"kubernetes.io/projected/09d8ee76-d883-42ab-871d-32be26a2f4f5-kube-api-access-2tck2\") pod \"ingress-canary-79v7b\" (UID: \"09d8ee76-d883-42ab-871d-32be26a2f4f5\") " pod="openshift-ingress-canary/ingress-canary-79v7b" Jul 06 20:59:11.897784 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.897747 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-79v7b" Jul 06 20:59:11.918199 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.918160 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-r9kt5"] Jul 06 20:59:11.930871 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.930844 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-r9kt5" Jul 06 20:59:11.931124 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.931100 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-r9kt5"] Jul 06 20:59:11.933384 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.933353 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-nx7px\"" Jul 06 20:59:11.933535 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.933449 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 06 20:59:11.933886 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.933852 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 06 20:59:11.973560 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:11.973526 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt2vt\" (UniqueName: \"kubernetes.io/projected/9067bb4c-c10e-49ae-bf8e-1809e833c8d9-kube-api-access-vt2vt\") pod \"migrator-d57d58cc9-r9kt5\" (UID: \"9067bb4c-c10e-49ae-bf8e-1809e833c8d9\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-r9kt5" Jul 06 20:59:12.074360 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:12.074313 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt2vt\" (UniqueName: \"kubernetes.io/projected/9067bb4c-c10e-49ae-bf8e-1809e833c8d9-kube-api-access-vt2vt\") pod \"migrator-d57d58cc9-r9kt5\" (UID: \"9067bb4c-c10e-49ae-bf8e-1809e833c8d9\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-r9kt5" Jul 06 20:59:12.076529 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:12.076484 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-79v7b"] Jul 06 20:59:12.082426 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:59:12.082386 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d8ee76_d883_42ab_871d_32be26a2f4f5.slice/crio-8e6d4b0dafb09b29b592ce0b6f6e490efda2420418e1b2d0d87e8742cc2c2b47 WatchSource:0}: Error finding container 8e6d4b0dafb09b29b592ce0b6f6e490efda2420418e1b2d0d87e8742cc2c2b47: Status 404 returned error can't find the container with id 8e6d4b0dafb09b29b592ce0b6f6e490efda2420418e1b2d0d87e8742cc2c2b47 Jul 06 20:59:12.085392 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:12.085365 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt2vt\" (UniqueName: \"kubernetes.io/projected/9067bb4c-c10e-49ae-bf8e-1809e833c8d9-kube-api-access-vt2vt\") pod \"migrator-d57d58cc9-r9kt5\" (UID: \"9067bb4c-c10e-49ae-bf8e-1809e833c8d9\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-r9kt5" Jul 06 20:59:12.174769 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:12.174671 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8886f6e5-67fc-4fd8-ad7b-95d98e776a1f-metrics-tls\") pod \"dns-default-xj2d6\" (UID: \"8886f6e5-67fc-4fd8-ad7b-95d98e776a1f\") " pod="openshift-dns/dns-default-xj2d6" Jul 06 20:59:12.174769 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:12.174746 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-registry-tls\") pod \"image-registry-645d5b8597-84426\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:12.175513 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:12.174830 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:59:12.175513 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:12.174849 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 20:59:12.175513 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:12.174859 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-645d5b8597-84426: secret "image-registry-tls" not found Jul 06 20:59:12.175513 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:12.174895 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8886f6e5-67fc-4fd8-ad7b-95d98e776a1f-metrics-tls podName:8886f6e5-67fc-4fd8-ad7b-95d98e776a1f nodeName:}" failed. No retries permitted until 2026-07-06 20:59:13.174880403 +0000 UTC m=+37.493103669 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8886f6e5-67fc-4fd8-ad7b-95d98e776a1f-metrics-tls") pod "dns-default-xj2d6" (UID: "8886f6e5-67fc-4fd8-ad7b-95d98e776a1f") : secret "dns-default-metrics-tls" not found Jul 06 20:59:12.175513 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:12.174910 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-registry-tls podName:ad03b367-ccbf-4b95-99f4-29e492b386ac nodeName:}" failed. No retries permitted until 2026-07-06 20:59:13.174903697 +0000 UTC m=+37.493126936 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-registry-tls") pod "image-registry-645d5b8597-84426" (UID: "ad03b367-ccbf-4b95-99f4-29e492b386ac") : secret "image-registry-tls" not found Jul 06 20:59:12.241627 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:12.241584 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-r9kt5" Jul 06 20:59:12.275797 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:12.275625 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7013baf3-f119-4c87-9c16-a9972cb124a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xq4l4\" (UID: \"7013baf3-f119-4c87-9c16-a9972cb124a8\") " pod="openshift-insights/insights-runtime-extractor-xq4l4" Jul 06 20:59:12.275797 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:12.275669 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7013baf3-f119-4c87-9c16-a9972cb124a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xq4l4\" (UID: \"7013baf3-f119-4c87-9c16-a9972cb124a8\") " pod="openshift-insights/insights-runtime-extractor-xq4l4" Jul 06 20:59:12.275797 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:12.275785 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 20:59:12.275950 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:12.275798 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 20:59:12.275950 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:12.275861 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7013baf3-f119-4c87-9c16-a9972cb124a8-insights-runtime-extractor-tls podName:7013baf3-f119-4c87-9c16-a9972cb124a8 nodeName:}" failed. No retries permitted until 2026-07-06 20:59:13.275840724 +0000 UTC m=+37.594063962 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7013baf3-f119-4c87-9c16-a9972cb124a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xq4l4" (UID: "7013baf3-f119-4c87-9c16-a9972cb124a8") : secret "insights-runtime-extractor-tls" not found Jul 06 20:59:12.275950 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:12.275880 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7013baf3-f119-4c87-9c16-a9972cb124a8-kube-rbac-proxy-cm podName:7013baf3-f119-4c87-9c16-a9972cb124a8 nodeName:}" failed. No retries permitted until 2026-07-06 20:59:13.275870315 +0000 UTC m=+37.594093559 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7013baf3-f119-4c87-9c16-a9972cb124a8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xq4l4" (UID: "7013baf3-f119-4c87-9c16-a9972cb124a8") : configmap "kube-rbac-proxy" not found Jul 06 20:59:12.337279 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:12.336826 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-79v7b" event={"ID":"09d8ee76-d883-42ab-871d-32be26a2f4f5","Type":"ContainerStarted","Data":"8e6d4b0dafb09b29b592ce0b6f6e490efda2420418e1b2d0d87e8742cc2c2b47"} Jul 06 20:59:12.406664 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:12.406631 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9mn4b_826779c0-bcd6-4c50-9ec1-0b62c9122657/dns-node-resolver/0.log" Jul 06 20:59:12.434093 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:12.433976 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-r9kt5"] Jul 06 20:59:12.437524 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:59:12.437495 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9067bb4c_c10e_49ae_bf8e_1809e833c8d9.slice/crio-3f267e1f96f0d4aaa077f214e1b625fb990d3e4493a17d99dff35d6cce27e6c7 WatchSource:0}: Error finding container 3f267e1f96f0d4aaa077f214e1b625fb990d3e4493a17d99dff35d6cce27e6c7: Status 404 returned error can't find the container with id 3f267e1f96f0d4aaa077f214e1b625fb990d3e4493a17d99dff35d6cce27e6c7 Jul 06 20:59:13.162629 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:13.162595 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:59:13.163424 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:13.162901 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:59:13.163424 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:13.163205 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:59:13.173350 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:13.173319 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 20:59:13.173718 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:13.173699 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 20:59:13.174304 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:13.174252 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7wh5f\"" Jul 06 20:59:13.174419 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:13.174397 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2tvmd\"" Jul 06 20:59:13.174492 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:13.174475 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 20:59:13.174662 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:13.174644 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 20:59:13.183787 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:13.183749 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-registry-tls\") pod \"image-registry-645d5b8597-84426\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:13.184251 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:13.183864 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8886f6e5-67fc-4fd8-ad7b-95d98e776a1f-metrics-tls\") pod \"dns-default-xj2d6\" (UID: \"8886f6e5-67fc-4fd8-ad7b-95d98e776a1f\") " pod="openshift-dns/dns-default-xj2d6" Jul 06 20:59:13.184251 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:13.184042 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:59:13.184251 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:13.184107 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8886f6e5-67fc-4fd8-ad7b-95d98e776a1f-metrics-tls podName:8886f6e5-67fc-4fd8-ad7b-95d98e776a1f nodeName:}" failed. No retries permitted until 2026-07-06 20:59:15.184086382 +0000 UTC m=+39.502309640 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8886f6e5-67fc-4fd8-ad7b-95d98e776a1f-metrics-tls") pod "dns-default-xj2d6" (UID: "8886f6e5-67fc-4fd8-ad7b-95d98e776a1f") : secret "dns-default-metrics-tls" not found Jul 06 20:59:13.184531 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:13.184046 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 20:59:13.184531 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:13.184310 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-645d5b8597-84426: secret "image-registry-tls" not found Jul 06 20:59:13.184531 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:13.184381 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-registry-tls podName:ad03b367-ccbf-4b95-99f4-29e492b386ac nodeName:}" failed. No retries permitted until 2026-07-06 20:59:15.184364805 +0000 UTC m=+39.502588060 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-registry-tls") pod "image-registry-645d5b8597-84426" (UID: "ad03b367-ccbf-4b95-99f4-29e492b386ac") : secret "image-registry-tls" not found Jul 06 20:59:13.284774 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:13.284743 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7013baf3-f119-4c87-9c16-a9972cb124a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xq4l4\" (UID: \"7013baf3-f119-4c87-9c16-a9972cb124a8\") " pod="openshift-insights/insights-runtime-extractor-xq4l4" Jul 06 20:59:13.284774 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:13.284779 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7013baf3-f119-4c87-9c16-a9972cb124a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xq4l4\" (UID: \"7013baf3-f119-4c87-9c16-a9972cb124a8\") " pod="openshift-insights/insights-runtime-extractor-xq4l4" Jul 06 20:59:13.285047 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:13.284884 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 20:59:13.285047 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:13.284933 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7013baf3-f119-4c87-9c16-a9972cb124a8-insights-runtime-extractor-tls podName:7013baf3-f119-4c87-9c16-a9972cb124a8 nodeName:}" failed. No retries permitted until 2026-07-06 20:59:15.284918875 +0000 UTC m=+39.603142113 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7013baf3-f119-4c87-9c16-a9972cb124a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xq4l4" (UID: "7013baf3-f119-4c87-9c16-a9972cb124a8") : secret "insights-runtime-extractor-tls" not found Jul 06 20:59:13.285168 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:13.285129 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 20:59:13.285325 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:13.285207 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7013baf3-f119-4c87-9c16-a9972cb124a8-kube-rbac-proxy-cm podName:7013baf3-f119-4c87-9c16-a9972cb124a8 nodeName:}" failed. No retries permitted until 2026-07-06 20:59:15.285188077 +0000 UTC m=+39.603411321 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7013baf3-f119-4c87-9c16-a9972cb124a8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xq4l4" (UID: "7013baf3-f119-4c87-9c16-a9972cb124a8") : configmap "kube-rbac-proxy" not found Jul 06 20:59:13.341075 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:13.341031 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-r9kt5" event={"ID":"9067bb4c-c10e-49ae-bf8e-1809e833c8d9","Type":"ContainerStarted","Data":"3f267e1f96f0d4aaa077f214e1b625fb990d3e4493a17d99dff35d6cce27e6c7"} Jul 06 20:59:13.569762 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:13.569646 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dmgkn_a9511f0e-6c1f-431c-8419-c1adc71f5520/node-ca/0.log" Jul 06 20:59:14.462716 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:14.462677 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-b7dxx"] Jul 06 20:59:14.470460 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:14.470327 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-b7dxx" Jul 06 20:59:14.472594 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:14.472558 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-b7dxx"] Jul 06 20:59:14.473274 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:14.473246 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 06 20:59:14.473597 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:14.473509 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 06 20:59:14.474048 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:14.473995 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 06 20:59:14.474410 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:14.474279 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-n4dcf\"" Jul 06 20:59:14.474549 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:14.474514 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 06 20:59:14.494913 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:14.494877 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/46823399-0495-40d1-b619-c6f9c2ffe142-signing-cabundle\") pod \"service-ca-758f599c5-b7dxx\" (UID: \"46823399-0495-40d1-b619-c6f9c2ffe142\") " pod="openshift-service-ca/service-ca-758f599c5-b7dxx" Jul 06 20:59:14.495116 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:14.494935 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/46823399-0495-40d1-b619-c6f9c2ffe142-signing-key\") pod \"service-ca-758f599c5-b7dxx\" (UID: \"46823399-0495-40d1-b619-c6f9c2ffe142\") " pod="openshift-service-ca/service-ca-758f599c5-b7dxx" Jul 06 20:59:14.495116 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:14.495068 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km6fr\" (UniqueName: \"kubernetes.io/projected/46823399-0495-40d1-b619-c6f9c2ffe142-kube-api-access-km6fr\") pod \"service-ca-758f599c5-b7dxx\" (UID: \"46823399-0495-40d1-b619-c6f9c2ffe142\") " pod="openshift-service-ca/service-ca-758f599c5-b7dxx" Jul 06 20:59:14.595933 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:14.595888 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km6fr\" (UniqueName: \"kubernetes.io/projected/46823399-0495-40d1-b619-c6f9c2ffe142-kube-api-access-km6fr\") pod \"service-ca-758f599c5-b7dxx\" (UID: \"46823399-0495-40d1-b619-c6f9c2ffe142\") " pod="openshift-service-ca/service-ca-758f599c5-b7dxx" Jul 06 20:59:14.596147 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:14.596061 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/46823399-0495-40d1-b619-c6f9c2ffe142-signing-cabundle\") pod \"service-ca-758f599c5-b7dxx\" (UID: \"46823399-0495-40d1-b619-c6f9c2ffe142\") " pod="openshift-service-ca/service-ca-758f599c5-b7dxx" Jul 06 20:59:14.596147 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:14.596101 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/46823399-0495-40d1-b619-c6f9c2ffe142-signing-key\") pod \"service-ca-758f599c5-b7dxx\" (UID: \"46823399-0495-40d1-b619-c6f9c2ffe142\") " pod="openshift-service-ca/service-ca-758f599c5-b7dxx" Jul 06 20:59:14.597255 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:14.597096 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/46823399-0495-40d1-b619-c6f9c2ffe142-signing-cabundle\") pod \"service-ca-758f599c5-b7dxx\" (UID: \"46823399-0495-40d1-b619-c6f9c2ffe142\") " pod="openshift-service-ca/service-ca-758f599c5-b7dxx" Jul 06 20:59:14.600430 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:14.600390 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/46823399-0495-40d1-b619-c6f9c2ffe142-signing-key\") pod \"service-ca-758f599c5-b7dxx\" (UID: \"46823399-0495-40d1-b619-c6f9c2ffe142\") " pod="openshift-service-ca/service-ca-758f599c5-b7dxx" Jul 06 20:59:14.605799 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:14.605758 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km6fr\" (UniqueName: \"kubernetes.io/projected/46823399-0495-40d1-b619-c6f9c2ffe142-kube-api-access-km6fr\") pod \"service-ca-758f599c5-b7dxx\" (UID: \"46823399-0495-40d1-b619-c6f9c2ffe142\") " pod="openshift-service-ca/service-ca-758f599c5-b7dxx" Jul 06 20:59:14.783429 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:14.783345 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-b7dxx" Jul 06 20:59:15.201096 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:15.201059 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-registry-tls\") pod \"image-registry-645d5b8597-84426\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:15.201297 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:15.201170 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8886f6e5-67fc-4fd8-ad7b-95d98e776a1f-metrics-tls\") pod \"dns-default-xj2d6\" (UID: \"8886f6e5-67fc-4fd8-ad7b-95d98e776a1f\") " pod="openshift-dns/dns-default-xj2d6" Jul 06 20:59:15.201297 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:15.201246 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 20:59:15.201297 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:15.201273 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-645d5b8597-84426: secret "image-registry-tls" not found Jul 06 20:59:15.201435 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:15.201303 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:59:15.201435 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:15.201342 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-registry-tls podName:ad03b367-ccbf-4b95-99f4-29e492b386ac nodeName:}" failed. No retries permitted until 2026-07-06 20:59:19.201325419 +0000 UTC m=+43.519548673 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-registry-tls") pod "image-registry-645d5b8597-84426" (UID: "ad03b367-ccbf-4b95-99f4-29e492b386ac") : secret "image-registry-tls" not found Jul 06 20:59:15.201435 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:15.201356 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8886f6e5-67fc-4fd8-ad7b-95d98e776a1f-metrics-tls podName:8886f6e5-67fc-4fd8-ad7b-95d98e776a1f nodeName:}" failed. No retries permitted until 2026-07-06 20:59:19.201350595 +0000 UTC m=+43.519573833 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8886f6e5-67fc-4fd8-ad7b-95d98e776a1f-metrics-tls") pod "dns-default-xj2d6" (UID: "8886f6e5-67fc-4fd8-ad7b-95d98e776a1f") : secret "dns-default-metrics-tls" not found Jul 06 20:59:15.301625 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:15.301593 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7013baf3-f119-4c87-9c16-a9972cb124a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xq4l4\" (UID: \"7013baf3-f119-4c87-9c16-a9972cb124a8\") " pod="openshift-insights/insights-runtime-extractor-xq4l4" Jul 06 20:59:15.301625 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:15.301628 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7013baf3-f119-4c87-9c16-a9972cb124a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xq4l4\" (UID: \"7013baf3-f119-4c87-9c16-a9972cb124a8\") " pod="openshift-insights/insights-runtime-extractor-xq4l4" Jul 06 20:59:15.301877 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:15.301699 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 20:59:15.301877 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:15.301735 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 20:59:15.301877 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:15.301768 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7013baf3-f119-4c87-9c16-a9972cb124a8-kube-rbac-proxy-cm podName:7013baf3-f119-4c87-9c16-a9972cb124a8 nodeName:}" failed. No retries permitted until 2026-07-06 20:59:19.301748194 +0000 UTC m=+43.619971455 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7013baf3-f119-4c87-9c16-a9972cb124a8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xq4l4" (UID: "7013baf3-f119-4c87-9c16-a9972cb124a8") : configmap "kube-rbac-proxy" not found Jul 06 20:59:15.301877 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:15.301786 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7013baf3-f119-4c87-9c16-a9972cb124a8-insights-runtime-extractor-tls podName:7013baf3-f119-4c87-9c16-a9972cb124a8 nodeName:}" failed. No retries permitted until 2026-07-06 20:59:19.30177912 +0000 UTC m=+43.620002358 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7013baf3-f119-4c87-9c16-a9972cb124a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xq4l4" (UID: "7013baf3-f119-4c87-9c16-a9972cb124a8") : secret "insights-runtime-extractor-tls" not found Jul 06 20:59:17.163451 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:17.163255 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-b7dxx"] Jul 06 20:59:17.166040 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:59:17.165991 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46823399_0495_40d1_b619_c6f9c2ffe142.slice/crio-ab080736a86591bf84d5ea5aa85fc23fef07b9194a06b4f4fcac93f76f8c12ef WatchSource:0}: Error finding container ab080736a86591bf84d5ea5aa85fc23fef07b9194a06b4f4fcac93f76f8c12ef: Status 404 returned error can't find the container with id ab080736a86591bf84d5ea5aa85fc23fef07b9194a06b4f4fcac93f76f8c12ef Jul 06 20:59:17.354089 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:17.354053 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-79v7b" event={"ID":"09d8ee76-d883-42ab-871d-32be26a2f4f5","Type":"ContainerStarted","Data":"204ce90925bd246461e50c56f74aa41d94da27e79d14b747064fb1c9968b7125"} Jul 06 20:59:17.355786 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:17.355754 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-r9kt5" event={"ID":"9067bb4c-c10e-49ae-bf8e-1809e833c8d9","Type":"ContainerStarted","Data":"21240d43d40535af279a387aba7cb8efd6912ddb7a5d3179187d0cc4f883d975"} Jul 06 20:59:17.355944 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:17.355815 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-r9kt5" event={"ID":"9067bb4c-c10e-49ae-bf8e-1809e833c8d9","Type":"ContainerStarted","Data":"07872dba4d528f395e17f0453ccf7ee49af89f067ecae3b05b3b851b2730e46e"} Jul 06 20:59:17.356909 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:17.356875 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-b7dxx" event={"ID":"46823399-0495-40d1-b619-c6f9c2ffe142","Type":"ContainerStarted","Data":"ab080736a86591bf84d5ea5aa85fc23fef07b9194a06b4f4fcac93f76f8c12ef"} Jul 06 20:59:17.359755 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:17.359723 2577 generic.go:356] "Generic (PLEG): container finished" podID="fcb1b646-77b2-4499-a4c7-4cfabfe3dcca" containerID="61c94356458920ccdc03f3d8ccbbf546280a66d181b2acaec6f09373d25ec6b2" exitCode=0 Jul 06 20:59:17.359907 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:17.359757 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7sdw" event={"ID":"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca","Type":"ContainerDied","Data":"61c94356458920ccdc03f3d8ccbbf546280a66d181b2acaec6f09373d25ec6b2"} Jul 06 20:59:17.374533 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:17.374478 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-79v7b" podStartSLOduration=1.44184738 podStartE2EDuration="6.374459222s" totalImagesPullingTime="4.932611842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:59:17.373388663 +0000 UTC m=+41.691611923" watchObservedRunningTime="2026-07-06 20:59:17.374459222 +0000 UTC m=+41.692682482" Jul 06 20:59:17.391614 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:17.391532 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-r9kt5" podStartSLOduration=1.814197884 podStartE2EDuration="6.391509644s" totalImagesPullingTime="4.57731176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:59:17.390806408 +0000 UTC m=+41.709029669" watchObservedRunningTime="2026-07-06 20:59:17.391509644 +0000 UTC m=+41.709732906" Jul 06 20:59:18.367081 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:18.366978 2577 generic.go:356] "Generic (PLEG): container finished" podID="fcb1b646-77b2-4499-a4c7-4cfabfe3dcca" containerID="fdd539ff01be371b65111743fdcf183236678cff64c84943ce557e5db6bc1847" exitCode=0 Jul 06 20:59:18.367081 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:18.367047 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7sdw" event={"ID":"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca","Type":"ContainerDied","Data":"fdd539ff01be371b65111743fdcf183236678cff64c84943ce557e5db6bc1847"} Jul 06 20:59:19.234303 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:19.234261 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-registry-tls\") pod \"image-registry-645d5b8597-84426\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:19.234524 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:19.234389 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8886f6e5-67fc-4fd8-ad7b-95d98e776a1f-metrics-tls\") pod \"dns-default-xj2d6\" (UID: \"8886f6e5-67fc-4fd8-ad7b-95d98e776a1f\") " pod="openshift-dns/dns-default-xj2d6" Jul 06 20:59:19.234524 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:19.234444 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 20:59:19.234524 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:19.234468 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-645d5b8597-84426: secret "image-registry-tls" not found Jul 06 20:59:19.234524 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:19.234520 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 20:59:19.234735 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:19.234538 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-registry-tls podName:ad03b367-ccbf-4b95-99f4-29e492b386ac nodeName:}" failed. No retries permitted until 2026-07-06 20:59:27.23451697 +0000 UTC m=+51.552740229 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-registry-tls") pod "image-registry-645d5b8597-84426" (UID: "ad03b367-ccbf-4b95-99f4-29e492b386ac") : secret "image-registry-tls" not found Jul 06 20:59:19.234735 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:19.234573 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8886f6e5-67fc-4fd8-ad7b-95d98e776a1f-metrics-tls podName:8886f6e5-67fc-4fd8-ad7b-95d98e776a1f nodeName:}" failed. No retries permitted until 2026-07-06 20:59:27.234558414 +0000 UTC m=+51.552781672 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8886f6e5-67fc-4fd8-ad7b-95d98e776a1f-metrics-tls") pod "dns-default-xj2d6" (UID: "8886f6e5-67fc-4fd8-ad7b-95d98e776a1f") : secret "dns-default-metrics-tls" not found Jul 06 20:59:19.335232 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:19.335177 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7013baf3-f119-4c87-9c16-a9972cb124a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xq4l4\" (UID: \"7013baf3-f119-4c87-9c16-a9972cb124a8\") " pod="openshift-insights/insights-runtime-extractor-xq4l4" Jul 06 20:59:19.335232 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:19.335223 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7013baf3-f119-4c87-9c16-a9972cb124a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xq4l4\" (UID: \"7013baf3-f119-4c87-9c16-a9972cb124a8\") " pod="openshift-insights/insights-runtime-extractor-xq4l4" Jul 06 20:59:19.335430 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:19.335321 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 20:59:19.335430 ip-10-0-136-207 kubenswrapper[2577]: E0706 20:59:19.335375 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7013baf3-f119-4c87-9c16-a9972cb124a8-insights-runtime-extractor-tls podName:7013baf3-f119-4c87-9c16-a9972cb124a8 nodeName:}" failed. No retries permitted until 2026-07-06 20:59:27.335361316 +0000 UTC m=+51.653584556 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7013baf3-f119-4c87-9c16-a9972cb124a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xq4l4" (UID: "7013baf3-f119-4c87-9c16-a9972cb124a8") : secret "insights-runtime-extractor-tls" not found Jul 06 20:59:19.335725 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:19.335707 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7013baf3-f119-4c87-9c16-a9972cb124a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xq4l4\" (UID: \"7013baf3-f119-4c87-9c16-a9972cb124a8\") " pod="openshift-insights/insights-runtime-extractor-xq4l4" Jul 06 20:59:19.373041 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:19.372992 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7sdw" event={"ID":"fcb1b646-77b2-4499-a4c7-4cfabfe3dcca","Type":"ContainerStarted","Data":"dff4cc0c357e751008970ded2187f5ac667315e94c6bc9888d4ff372b73c976d"} Jul 06 20:59:19.418768 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:19.418710 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-v7sdw" podStartSLOduration=8.941633297 podStartE2EDuration="43.418691153s" totalImagesPullingTime="34.477057856s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:58:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:59:19.418214634 +0000 UTC m=+43.736437875" watchObservedRunningTime="2026-07-06 20:59:19.418691153 +0000 UTC m=+43.736914417" Jul 06 20:59:20.377156 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:20.377061 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-b7dxx" event={"ID":"46823399-0495-40d1-b619-c6f9c2ffe142","Type":"ContainerStarted","Data":"22557e58af23988a1695ad1b0a2f885f62dd2c6579bfc280ec549ddf66a89894"} Jul 06 20:59:20.401409 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:20.401354 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-b7dxx" podStartSLOduration=3.846142491 podStartE2EDuration="6.401336846s" totalImagesPullingTime="2.555194355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:59:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:59:20.401055867 +0000 UTC m=+44.719279146" watchObservedRunningTime="2026-07-06 20:59:20.401336846 +0000 UTC m=+44.719560146" Jul 06 20:59:27.302267 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:27.302227 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8886f6e5-67fc-4fd8-ad7b-95d98e776a1f-metrics-tls\") pod \"dns-default-xj2d6\" (UID: \"8886f6e5-67fc-4fd8-ad7b-95d98e776a1f\") " pod="openshift-dns/dns-default-xj2d6" Jul 06 20:59:27.302667 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:27.302302 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-registry-tls\") pod \"image-registry-645d5b8597-84426\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:27.305151 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:27.305123 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8886f6e5-67fc-4fd8-ad7b-95d98e776a1f-metrics-tls\") pod \"dns-default-xj2d6\" (UID: \"8886f6e5-67fc-4fd8-ad7b-95d98e776a1f\") " pod="openshift-dns/dns-default-xj2d6" Jul 06 20:59:27.305259 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:27.305199 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-registry-tls\") pod \"image-registry-645d5b8597-84426\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:27.403200 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:27.403160 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7013baf3-f119-4c87-9c16-a9972cb124a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xq4l4\" (UID: \"7013baf3-f119-4c87-9c16-a9972cb124a8\") " pod="openshift-insights/insights-runtime-extractor-xq4l4" Jul 06 20:59:27.405637 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:27.405613 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7013baf3-f119-4c87-9c16-a9972cb124a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xq4l4\" (UID: \"7013baf3-f119-4c87-9c16-a9972cb124a8\") " pod="openshift-insights/insights-runtime-extractor-xq4l4" Jul 06 20:59:27.438626 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:27.438579 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:27.453443 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:27.453406 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xj2d6" Jul 06 20:59:27.477559 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:27.477516 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xq4l4" Jul 06 20:59:27.607649 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:27.607626 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-645d5b8597-84426"] Jul 06 20:59:27.610833 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:59:27.610801 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad03b367_ccbf_4b95_99f4_29e492b386ac.slice/crio-feb9fa94368634654bc9db29bec694b09569ab0b9e86891cb1fb970c393231e9 WatchSource:0}: Error finding container feb9fa94368634654bc9db29bec694b09569ab0b9e86891cb1fb970c393231e9: Status 404 returned error can't find the container with id feb9fa94368634654bc9db29bec694b09569ab0b9e86891cb1fb970c393231e9 Jul 06 20:59:27.627027 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:27.626986 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xj2d6"] Jul 06 20:59:27.627468 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:59:27.627444 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8886f6e5_67fc_4fd8_ad7b_95d98e776a1f.slice/crio-424d898aa931d3f645d75b98e434a2a7bee4f1aa9bf96ee1ae56e41fde56eaa1 WatchSource:0}: Error finding container 424d898aa931d3f645d75b98e434a2a7bee4f1aa9bf96ee1ae56e41fde56eaa1: Status 404 returned error can't find the container with id 424d898aa931d3f645d75b98e434a2a7bee4f1aa9bf96ee1ae56e41fde56eaa1 Jul 06 20:59:27.652147 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:27.652113 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xq4l4"] Jul 06 20:59:27.655518 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:59:27.655483 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7013baf3_f119_4c87_9c16_a9972cb124a8.slice/crio-df06f0c342d84030422e21e4e8386b5a85f20828b5d9db5e09dc52cb66cf67ed WatchSource:0}: Error finding container df06f0c342d84030422e21e4e8386b5a85f20828b5d9db5e09dc52cb66cf67ed: Status 404 returned error can't find the container with id df06f0c342d84030422e21e4e8386b5a85f20828b5d9db5e09dc52cb66cf67ed Jul 06 20:59:28.403699 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:28.403536 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xq4l4" event={"ID":"7013baf3-f119-4c87-9c16-a9972cb124a8","Type":"ContainerStarted","Data":"df06f0c342d84030422e21e4e8386b5a85f20828b5d9db5e09dc52cb66cf67ed"} Jul 06 20:59:28.406896 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:28.406857 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xj2d6" event={"ID":"8886f6e5-67fc-4fd8-ad7b-95d98e776a1f","Type":"ContainerStarted","Data":"424d898aa931d3f645d75b98e434a2a7bee4f1aa9bf96ee1ae56e41fde56eaa1"} Jul 06 20:59:28.410698 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:28.410603 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-645d5b8597-84426" event={"ID":"ad03b367-ccbf-4b95-99f4-29e492b386ac","Type":"ContainerStarted","Data":"61281e7e49957bdcf223f919fdc02d5eacbd8147fd15e577b7b38a4452958c29"} Jul 06 20:59:28.410698 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:28.410665 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-645d5b8597-84426" event={"ID":"ad03b367-ccbf-4b95-99f4-29e492b386ac","Type":"ContainerStarted","Data":"feb9fa94368634654bc9db29bec694b09569ab0b9e86891cb1fb970c393231e9"} Jul 06 20:59:28.410998 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:28.410713 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:28.436061 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:28.435688 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-645d5b8597-84426" podStartSLOduration=38.435661279 podStartE2EDuration="38.435661279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:58:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:59:28.434493421 +0000 UTC m=+52.752716684" watchObservedRunningTime="2026-07-06 20:59:28.435661279 +0000 UTC m=+52.753884542" Jul 06 20:59:32.428190 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:32.428149 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xq4l4" event={"ID":"7013baf3-f119-4c87-9c16-a9972cb124a8","Type":"ContainerStarted","Data":"f1397b883c40d4fcf7c1caab99cc4a72088d67e72e97e38ed2cd3a8d88c2db23"} Jul 06 20:59:32.430609 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:32.430573 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xj2d6" event={"ID":"8886f6e5-67fc-4fd8-ad7b-95d98e776a1f","Type":"ContainerStarted","Data":"a96c7def731406db716449f79ba60af0d3dd1476680169c5cc1ee3008fb012df"} Jul 06 20:59:32.430779 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:32.430636 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xj2d6" event={"ID":"8886f6e5-67fc-4fd8-ad7b-95d98e776a1f","Type":"ContainerStarted","Data":"b2a46bcb5f07c082d7aa20c35c67a2eeb5d9a07306e4bd30bcca50c319a23753"} Jul 06 20:59:32.430779 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:32.430723 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xj2d6" Jul 06 20:59:32.458422 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:32.458359 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xj2d6" podStartSLOduration=17.142865648 podStartE2EDuration="21.458336347s" totalImagesPullingTime="4.315470699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:59:32.457425195 +0000 UTC m=+56.775648478" watchObservedRunningTime="2026-07-06 20:59:32.458336347 +0000 UTC m=+56.776559610" Jul 06 20:59:33.440063 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:33.440025 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xq4l4" event={"ID":"7013baf3-f119-4c87-9c16-a9972cb124a8","Type":"ContainerStarted","Data":"37e0945438e7ee765918e0409886fa56e56d28edb64e077ca91979ebaffe615c"} Jul 06 20:59:35.454509 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:35.454415 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xq4l4" event={"ID":"7013baf3-f119-4c87-9c16-a9972cb124a8","Type":"ContainerStarted","Data":"6e6422db984bd9473225c29cf2f1d11007069bb07656f60161c39710718234ee"} Jul 06 20:59:35.474336 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:35.474277 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xq4l4" podStartSLOduration=17.100316643 podStartE2EDuration="24.474258791s" totalImagesPullingTime="7.373942148s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:59:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:59:35.473703727 +0000 UTC m=+59.791927014" watchObservedRunningTime="2026-07-06 20:59:35.474258791 +0000 UTC m=+59.792482089" Jul 06 20:59:37.318775 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:37.318746 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qjs2n" Jul 06 20:59:38.679613 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:38.679579 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-m5kns"] Jul 06 20:59:38.705965 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:38.705929 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-m5kns"] Jul 06 20:59:38.706265 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:38.706060 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-m5kns" Jul 06 20:59:38.709483 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:38.709449 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-sdxtg\"" Jul 06 20:59:38.709483 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:38.709464 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 06 20:59:38.709710 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:38.709472 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 06 20:59:38.745423 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:38.745386 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-645d5b8597-84426"] Jul 06 20:59:38.792522 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:38.792485 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-j7pvh"] Jul 06 20:59:38.804826 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:38.804793 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-j7pvh" Jul 06 20:59:38.806836 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:38.806804 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7c85aee9-0db9-43f1-8011-f04079c0bce8-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-m5kns\" (UID: \"7c85aee9-0db9-43f1-8011-f04079c0bce8\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-m5kns" Jul 06 20:59:38.806974 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:38.806847 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7c85aee9-0db9-43f1-8011-f04079c0bce8-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-m5kns\" (UID: \"7c85aee9-0db9-43f1-8011-f04079c0bce8\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-m5kns" Jul 06 20:59:38.807215 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:38.807192 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 20:59:38.807711 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:38.807688 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-j7pvh"] Jul 06 20:59:38.807820 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:38.807746 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 20:59:38.807883 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:38.807868 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-thpxh\"" Jul 06 20:59:38.907975 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:38.907940 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht5r7\" (UniqueName: \"kubernetes.io/projected/e7359088-7a08-466a-8bac-0cd67affc60f-kube-api-access-ht5r7\") pod \"downloads-685bfb5f64-j7pvh\" (UID: \"e7359088-7a08-466a-8bac-0cd67affc60f\") " pod="openshift-console/downloads-685bfb5f64-j7pvh" Jul 06 20:59:38.908155 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:38.908058 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7c85aee9-0db9-43f1-8011-f04079c0bce8-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-m5kns\" (UID: \"7c85aee9-0db9-43f1-8011-f04079c0bce8\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-m5kns" Jul 06 20:59:38.908155 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:38.908096 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7c85aee9-0db9-43f1-8011-f04079c0bce8-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-m5kns\" (UID: \"7c85aee9-0db9-43f1-8011-f04079c0bce8\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-m5kns" Jul 06 20:59:38.908841 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:38.908822 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7c85aee9-0db9-43f1-8011-f04079c0bce8-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-m5kns\" (UID: \"7c85aee9-0db9-43f1-8011-f04079c0bce8\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-m5kns" Jul 06 20:59:38.910777 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:38.910752 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7c85aee9-0db9-43f1-8011-f04079c0bce8-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-m5kns\" (UID: \"7c85aee9-0db9-43f1-8011-f04079c0bce8\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-m5kns" Jul 06 20:59:39.009216 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:39.009126 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht5r7\" (UniqueName: \"kubernetes.io/projected/e7359088-7a08-466a-8bac-0cd67affc60f-kube-api-access-ht5r7\") pod \"downloads-685bfb5f64-j7pvh\" (UID: \"e7359088-7a08-466a-8bac-0cd67affc60f\") " pod="openshift-console/downloads-685bfb5f64-j7pvh" Jul 06 20:59:39.016101 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:39.016065 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-m5kns" Jul 06 20:59:39.033775 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:39.033741 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht5r7\" (UniqueName: \"kubernetes.io/projected/e7359088-7a08-466a-8bac-0cd67affc60f-kube-api-access-ht5r7\") pod \"downloads-685bfb5f64-j7pvh\" (UID: \"e7359088-7a08-466a-8bac-0cd67affc60f\") " pod="openshift-console/downloads-685bfb5f64-j7pvh" Jul 06 20:59:39.115346 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:39.115150 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-j7pvh" Jul 06 20:59:39.187863 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:39.187763 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-m5kns"] Jul 06 20:59:39.192215 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:59:39.192185 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c85aee9_0db9_43f1_8011_f04079c0bce8.slice/crio-85707dee6f7859b21ebd3c8c34cbc629349a5809f678fd68d4a021d753bbd985 WatchSource:0}: Error finding container 85707dee6f7859b21ebd3c8c34cbc629349a5809f678fd68d4a021d753bbd985: Status 404 returned error can't find the container with id 85707dee6f7859b21ebd3c8c34cbc629349a5809f678fd68d4a021d753bbd985 Jul 06 20:59:39.298381 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:39.298264 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-j7pvh"] Jul 06 20:59:39.300913 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:59:39.300880 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7359088_7a08_466a_8bac_0cd67affc60f.slice/crio-53b3a60ebd085ea3fa085f062cdcbe300376c90a0e5a4c4c9593346354495777 WatchSource:0}: Error finding container 53b3a60ebd085ea3fa085f062cdcbe300376c90a0e5a4c4c9593346354495777: Status 404 returned error can't find the container with id 53b3a60ebd085ea3fa085f062cdcbe300376c90a0e5a4c4c9593346354495777 Jul 06 20:59:39.468097 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:39.468058 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-m5kns" event={"ID":"7c85aee9-0db9-43f1-8011-f04079c0bce8","Type":"ContainerStarted","Data":"85707dee6f7859b21ebd3c8c34cbc629349a5809f678fd68d4a021d753bbd985"} Jul 06 20:59:39.469165 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:39.469130 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-j7pvh" event={"ID":"e7359088-7a08-466a-8bac-0cd67affc60f","Type":"ContainerStarted","Data":"53b3a60ebd085ea3fa085f062cdcbe300376c90a0e5a4c4c9593346354495777"} Jul 06 20:59:40.475352 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:40.475306 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-m5kns" event={"ID":"7c85aee9-0db9-43f1-8011-f04079c0bce8","Type":"ContainerStarted","Data":"7d687b0b1c8fecc072e0e5f2dcd23c8986119a3205468c9446d82fd81b054a8c"} Jul 06 20:59:40.502493 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:40.502434 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-m5kns" podStartSLOduration=1.316006359 podStartE2EDuration="2.502414905s" totalImagesPullingTime="1.186408546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:59:40.501437672 +0000 UTC m=+64.819660933" watchObservedRunningTime="2026-07-06 20:59:40.502414905 +0000 UTC m=+64.820638165" Jul 06 20:59:40.927678 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:40.927639 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bb28f09-af91-446b-9f07-6a9a331b2bf5-metrics-certs\") pod \"network-metrics-daemon-qc5r5\" (UID: \"6bb28f09-af91-446b-9f07-6a9a331b2bf5\") " pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:59:40.927873 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:40.927694 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0f9a2ff-6f72-4374-a459-b4da598d3533-original-pull-secret\") pod \"global-pull-secret-syncer-dbnt9\" (UID: \"b0f9a2ff-6f72-4374-a459-b4da598d3533\") " pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:59:40.930807 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:40.930775 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 20:59:40.931593 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:40.931570 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 20:59:40.941382 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:40.941351 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bb28f09-af91-446b-9f07-6a9a331b2bf5-metrics-certs\") pod \"network-metrics-daemon-qc5r5\" (UID: \"6bb28f09-af91-446b-9f07-6a9a331b2bf5\") " pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:59:40.941541 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:40.941469 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0f9a2ff-6f72-4374-a459-b4da598d3533-original-pull-secret\") pod \"global-pull-secret-syncer-dbnt9\" (UID: \"b0f9a2ff-6f72-4374-a459-b4da598d3533\") " pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:59:41.028243 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:41.028202 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj6kq\" (UniqueName: \"kubernetes.io/projected/b3810ec4-264c-4e94-ba9e-ab262e657770-kube-api-access-lj6kq\") pod \"network-check-target-n8x9c\" (UID: \"b3810ec4-264c-4e94-ba9e-ab262e657770\") " pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:59:41.031180 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:41.031150 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 20:59:41.041500 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:41.041473 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 20:59:41.052574 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:41.052541 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj6kq\" (UniqueName: \"kubernetes.io/projected/b3810ec4-264c-4e94-ba9e-ab262e657770-kube-api-access-lj6kq\") pod \"network-check-target-n8x9c\" (UID: \"b3810ec4-264c-4e94-ba9e-ab262e657770\") " pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:59:41.079351 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:41.079312 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbnt9" Jul 06 20:59:41.092544 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:41.092516 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7wh5f\"" Jul 06 20:59:41.099997 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:41.099957 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qc5r5" Jul 06 20:59:41.100533 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:41.100466 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2tvmd\"" Jul 06 20:59:41.108697 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:41.108651 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:59:41.258992 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:41.258956 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dbnt9"] Jul 06 20:59:41.260381 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:59:41.260145 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f9a2ff_6f72_4374_a459_b4da598d3533.slice/crio-3649d7c8b6aa8c600c403eff163e6ef264275b801523bda8871927648a220051 WatchSource:0}: Error finding container 3649d7c8b6aa8c600c403eff163e6ef264275b801523bda8871927648a220051: Status 404 returned error can't find the container with id 3649d7c8b6aa8c600c403eff163e6ef264275b801523bda8871927648a220051 Jul 06 20:59:41.278872 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:41.278625 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qc5r5"] Jul 06 20:59:41.281527 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:59:41.281494 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb28f09_af91_446b_9f07_6a9a331b2bf5.slice/crio-e04a2bf69682e8267116dbaeee540c1f8af7884b3829a1c572045bdae785deb4 WatchSource:0}: Error finding container e04a2bf69682e8267116dbaeee540c1f8af7884b3829a1c572045bdae785deb4: Status 404 returned error can't find the container with id e04a2bf69682e8267116dbaeee540c1f8af7884b3829a1c572045bdae785deb4 Jul 06 20:59:41.299627 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:41.299489 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n8x9c"] Jul 06 20:59:41.302418 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:59:41.302386 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3810ec4_264c_4e94_ba9e_ab262e657770.slice/crio-dbf1cc95cb71948c7d5d380ae51113de659a17e41255bfdb1d03d9514101eb7a WatchSource:0}: Error finding container dbf1cc95cb71948c7d5d380ae51113de659a17e41255bfdb1d03d9514101eb7a: Status 404 returned error can't find the container with id dbf1cc95cb71948c7d5d380ae51113de659a17e41255bfdb1d03d9514101eb7a Jul 06 20:59:41.479428 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:41.479329 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qc5r5" event={"ID":"6bb28f09-af91-446b-9f07-6a9a331b2bf5","Type":"ContainerStarted","Data":"e04a2bf69682e8267116dbaeee540c1f8af7884b3829a1c572045bdae785deb4"} Jul 06 20:59:41.480456 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:41.480433 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dbnt9" event={"ID":"b0f9a2ff-6f72-4374-a459-b4da598d3533","Type":"ContainerStarted","Data":"3649d7c8b6aa8c600c403eff163e6ef264275b801523bda8871927648a220051"} Jul 06 20:59:41.481534 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:41.481506 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n8x9c" event={"ID":"b3810ec4-264c-4e94-ba9e-ab262e657770","Type":"ContainerStarted","Data":"dbf1cc95cb71948c7d5d380ae51113de659a17e41255bfdb1d03d9514101eb7a"} Jul 06 20:59:42.443439 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:42.442661 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xj2d6" Jul 06 20:59:43.497950 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:43.497637 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qc5r5" event={"ID":"6bb28f09-af91-446b-9f07-6a9a331b2bf5","Type":"ContainerStarted","Data":"60af0ca9676a93f4e3886d25d15ab6d4f65be04a91d85519c13c3e92e7b25d47"} Jul 06 20:59:43.497950 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:43.497706 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qc5r5" event={"ID":"6bb28f09-af91-446b-9f07-6a9a331b2bf5","Type":"ContainerStarted","Data":"50d384bb5870782b1f6b0c1422ef9f1a718e996513e03ad48f7d879a795422c1"} Jul 06 20:59:48.522317 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:48.522274 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dbnt9" event={"ID":"b0f9a2ff-6f72-4374-a459-b4da598d3533","Type":"ContainerStarted","Data":"fc7395cf07290ffa604358d5b581bb3fe242fa49241295d9263b6a88c22c8506"} Jul 06 20:59:48.524093 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:48.524055 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n8x9c" event={"ID":"b3810ec4-264c-4e94-ba9e-ab262e657770","Type":"ContainerStarted","Data":"371f01aa349b86d3328512db09822bb47a15e7b459cca61d6f866cc04633a924"} Jul 06 20:59:48.524253 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:48.524191 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 20:59:48.587256 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:48.587198 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qc5r5" podStartSLOduration=71.333753951 podStartE2EDuration="1m12.587172622s" totalImagesPullingTime="1.253418671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:59:43.52310305 +0000 UTC m=+67.841326310" watchObservedRunningTime="2026-07-06 20:59:48.587172622 +0000 UTC m=+72.905395883" Jul 06 20:59:48.587910 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:48.587845 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dbnt9" podStartSLOduration=66.228677051 podStartE2EDuration="1m12.587832408s" totalImagesPullingTime="6.359155357s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 20:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:59:48.587655015 +0000 UTC m=+72.905878275" watchObservedRunningTime="2026-07-06 20:59:48.587832408 +0000 UTC m=+72.906055667" Jul 06 20:59:48.624974 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:48.624827 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-n8x9c" podStartSLOduration=66.32121673 podStartE2EDuration="1m12.624804095s" totalImagesPullingTime="6.303587365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:59:48.623185894 +0000 UTC m=+72.941409153" watchObservedRunningTime="2026-07-06 20:59:48.624804095 +0000 UTC m=+72.943027358" Jul 06 20:59:48.752277 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:48.752165 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 20:59:58.564992 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:58.564899 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-j7pvh" event={"ID":"e7359088-7a08-466a-8bac-0cd67affc60f","Type":"ContainerStarted","Data":"403cfce564915ff3011d5f4ec291f3b7502c64bbd0ccba2ba90c78878ba85d4f"} Jul 06 20:59:58.565515 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:58.565236 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-j7pvh" Jul 06 20:59:58.578551 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:58.578515 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-j7pvh" Jul 06 20:59:58.586386 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:58.586290 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-j7pvh" podStartSLOduration=1.66976361 podStartE2EDuration="20.586270159s" totalImagesPullingTime="18.916506549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 20:59:58.586267436 +0000 UTC m=+82.904490723" watchObservedRunningTime="2026-07-06 20:59:58.586270159 +0000 UTC m=+82.904493421" Jul 06 20:59:59.453849 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.453809 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dhcnh"] Jul 06 20:59:59.482364 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.482322 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dhcnh"] Jul 06 20:59:59.482550 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.482391 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-785dd"] Jul 06 20:59:59.482550 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.482535 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dhcnh" Jul 06 20:59:59.485291 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.485264 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 06 20:59:59.485618 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.485591 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 20:59:59.485750 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.485687 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 06 20:59:59.485973 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.485831 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 20:59:59.486634 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.486614 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 06 20:59:59.486729 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.486623 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 20:59:59.487524 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.487502 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-hz8mh\"" Jul 06 20:59:59.504164 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.504127 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-785dd" Jul 06 20:59:59.511775 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.508938 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 20:59:59.511775 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.511085 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 20:59:59.511775 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.511149 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 20:59:59.518044 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.517754 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tj46g\"" Jul 06 20:59:59.580536 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.580494 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0ab3ede5-c596-4fab-a7db-683fa6c81ebc-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-dhcnh\" (UID: \"0ab3ede5-c596-4fab-a7db-683fa6c81ebc\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dhcnh" Jul 06 20:59:59.581052 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.580554 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/536941ec-7aff-42ca-b8d9-fba6959b98aa-root\") pod \"node-exporter-785dd\" (UID: \"536941ec-7aff-42ca-b8d9-fba6959b98aa\") " pod="openshift-monitoring/node-exporter-785dd" Jul 06 20:59:59.581052 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.580635 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/536941ec-7aff-42ca-b8d9-fba6959b98aa-sys\") pod \"node-exporter-785dd\" (UID: \"536941ec-7aff-42ca-b8d9-fba6959b98aa\") " pod="openshift-monitoring/node-exporter-785dd" Jul 06 20:59:59.581052 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.580674 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/536941ec-7aff-42ca-b8d9-fba6959b98aa-node-exporter-textfile\") pod \"node-exporter-785dd\" (UID: \"536941ec-7aff-42ca-b8d9-fba6959b98aa\") " pod="openshift-monitoring/node-exporter-785dd" Jul 06 20:59:59.581052 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.580705 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ab3ede5-c596-4fab-a7db-683fa6c81ebc-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-dhcnh\" (UID: \"0ab3ede5-c596-4fab-a7db-683fa6c81ebc\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dhcnh" Jul 06 20:59:59.581052 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.580738 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/536941ec-7aff-42ca-b8d9-fba6959b98aa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-785dd\" (UID: \"536941ec-7aff-42ca-b8d9-fba6959b98aa\") " pod="openshift-monitoring/node-exporter-785dd" Jul 06 20:59:59.581052 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.580766 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ab3ede5-c596-4fab-a7db-683fa6c81ebc-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-dhcnh\" (UID: \"0ab3ede5-c596-4fab-a7db-683fa6c81ebc\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dhcnh" Jul 06 20:59:59.581052 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.580790 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/536941ec-7aff-42ca-b8d9-fba6959b98aa-node-exporter-accelerators-collector-config\") pod \"node-exporter-785dd\" (UID: \"536941ec-7aff-42ca-b8d9-fba6959b98aa\") " pod="openshift-monitoring/node-exporter-785dd" Jul 06 20:59:59.581052 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.580828 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/536941ec-7aff-42ca-b8d9-fba6959b98aa-metrics-client-ca\") pod \"node-exporter-785dd\" (UID: \"536941ec-7aff-42ca-b8d9-fba6959b98aa\") " pod="openshift-monitoring/node-exporter-785dd" Jul 06 20:59:59.581052 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.580856 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfh45\" (UniqueName: \"kubernetes.io/projected/0ab3ede5-c596-4fab-a7db-683fa6c81ebc-kube-api-access-vfh45\") pod \"kube-state-metrics-7c9dbb8b7d-dhcnh\" (UID: \"0ab3ede5-c596-4fab-a7db-683fa6c81ebc\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dhcnh" Jul 06 20:59:59.581052 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.580887 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/536941ec-7aff-42ca-b8d9-fba6959b98aa-node-exporter-tls\") pod \"node-exporter-785dd\" (UID: \"536941ec-7aff-42ca-b8d9-fba6959b98aa\") " pod="openshift-monitoring/node-exporter-785dd" Jul 06 20:59:59.581052 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.580936 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0ab3ede5-c596-4fab-a7db-683fa6c81ebc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-dhcnh\" (UID: \"0ab3ede5-c596-4fab-a7db-683fa6c81ebc\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dhcnh" Jul 06 20:59:59.581052 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.580962 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/536941ec-7aff-42ca-b8d9-fba6959b98aa-node-exporter-wtmp\") pod \"node-exporter-785dd\" (UID: \"536941ec-7aff-42ca-b8d9-fba6959b98aa\") " pod="openshift-monitoring/node-exporter-785dd" Jul 06 20:59:59.581052 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.581026 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ab3ede5-c596-4fab-a7db-683fa6c81ebc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-dhcnh\" (UID: \"0ab3ede5-c596-4fab-a7db-683fa6c81ebc\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dhcnh" Jul 06 20:59:59.581700 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.581050 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4fvs\" (UniqueName: \"kubernetes.io/projected/536941ec-7aff-42ca-b8d9-fba6959b98aa-kube-api-access-v4fvs\") pod \"node-exporter-785dd\" (UID: \"536941ec-7aff-42ca-b8d9-fba6959b98aa\") " pod="openshift-monitoring/node-exporter-785dd" Jul 06 20:59:59.681462 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.681417 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0ab3ede5-c596-4fab-a7db-683fa6c81ebc-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-dhcnh\" (UID: \"0ab3ede5-c596-4fab-a7db-683fa6c81ebc\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dhcnh" Jul 06 20:59:59.681462 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.681471 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/536941ec-7aff-42ca-b8d9-fba6959b98aa-root\") pod \"node-exporter-785dd\" (UID: \"536941ec-7aff-42ca-b8d9-fba6959b98aa\") " pod="openshift-monitoring/node-exporter-785dd" Jul 06 20:59:59.681729 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.681499 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/536941ec-7aff-42ca-b8d9-fba6959b98aa-sys\") pod \"node-exporter-785dd\" (UID: \"536941ec-7aff-42ca-b8d9-fba6959b98aa\") " pod="openshift-monitoring/node-exporter-785dd" Jul 06 20:59:59.681729 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.681523 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/536941ec-7aff-42ca-b8d9-fba6959b98aa-node-exporter-textfile\") pod \"node-exporter-785dd\" (UID: \"536941ec-7aff-42ca-b8d9-fba6959b98aa\") " pod="openshift-monitoring/node-exporter-785dd" Jul 06 20:59:59.681729 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.681550 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ab3ede5-c596-4fab-a7db-683fa6c81ebc-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-dhcnh\" (UID: \"0ab3ede5-c596-4fab-a7db-683fa6c81ebc\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dhcnh" Jul 06 20:59:59.681729 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.681581 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/536941ec-7aff-42ca-b8d9-fba6959b98aa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-785dd\" (UID: \"536941ec-7aff-42ca-b8d9-fba6959b98aa\") " pod="openshift-monitoring/node-exporter-785dd" Jul 06 20:59:59.681729 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.681606 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ab3ede5-c596-4fab-a7db-683fa6c81ebc-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-dhcnh\" (UID: \"0ab3ede5-c596-4fab-a7db-683fa6c81ebc\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dhcnh" Jul 06 20:59:59.681729 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.681628 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/536941ec-7aff-42ca-b8d9-fba6959b98aa-node-exporter-accelerators-collector-config\") pod \"node-exporter-785dd\" (UID: \"536941ec-7aff-42ca-b8d9-fba6959b98aa\") " pod="openshift-monitoring/node-exporter-785dd" Jul 06 20:59:59.681729 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.681639 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/536941ec-7aff-42ca-b8d9-fba6959b98aa-sys\") pod \"node-exporter-785dd\" (UID: \"536941ec-7aff-42ca-b8d9-fba6959b98aa\") " pod="openshift-monitoring/node-exporter-785dd" Jul 06 20:59:59.681729 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.681700 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/536941ec-7aff-42ca-b8d9-fba6959b98aa-metrics-client-ca\") pod \"node-exporter-785dd\" (UID: \"536941ec-7aff-42ca-b8d9-fba6959b98aa\") " pod="openshift-monitoring/node-exporter-785dd" Jul 06 20:59:59.682143 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.681741 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfh45\" (UniqueName: \"kubernetes.io/projected/0ab3ede5-c596-4fab-a7db-683fa6c81ebc-kube-api-access-vfh45\") pod \"kube-state-metrics-7c9dbb8b7d-dhcnh\" (UID: \"0ab3ede5-c596-4fab-a7db-683fa6c81ebc\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dhcnh" Jul 06 20:59:59.682143 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.681780 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/536941ec-7aff-42ca-b8d9-fba6959b98aa-node-exporter-tls\") pod \"node-exporter-785dd\" (UID: \"536941ec-7aff-42ca-b8d9-fba6959b98aa\") " pod="openshift-monitoring/node-exporter-785dd" Jul 06 20:59:59.682143 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.681835 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0ab3ede5-c596-4fab-a7db-683fa6c81ebc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-dhcnh\" (UID: \"0ab3ede5-c596-4fab-a7db-683fa6c81ebc\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dhcnh" Jul 06 20:59:59.682143 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.681864 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/536941ec-7aff-42ca-b8d9-fba6959b98aa-node-exporter-wtmp\") pod \"node-exporter-785dd\" (UID: \"536941ec-7aff-42ca-b8d9-fba6959b98aa\") " pod="openshift-monitoring/node-exporter-785dd" Jul 06 20:59:59.682143 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.681877 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0ab3ede5-c596-4fab-a7db-683fa6c81ebc-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-dhcnh\" (UID: \"0ab3ede5-c596-4fab-a7db-683fa6c81ebc\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dhcnh" Jul 06 20:59:59.682143 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.681904 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ab3ede5-c596-4fab-a7db-683fa6c81ebc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-dhcnh\" (UID: \"0ab3ede5-c596-4fab-a7db-683fa6c81ebc\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dhcnh" Jul 06 20:59:59.682143 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.681928 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/536941ec-7aff-42ca-b8d9-fba6959b98aa-node-exporter-textfile\") pod \"node-exporter-785dd\" (UID: \"536941ec-7aff-42ca-b8d9-fba6959b98aa\") " pod="openshift-monitoring/node-exporter-785dd" Jul 06 20:59:59.682143 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.681933 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4fvs\" (UniqueName: \"kubernetes.io/projected/536941ec-7aff-42ca-b8d9-fba6959b98aa-kube-api-access-v4fvs\") pod \"node-exporter-785dd\" (UID: \"536941ec-7aff-42ca-b8d9-fba6959b98aa\") " pod="openshift-monitoring/node-exporter-785dd" Jul 06 20:59:59.682143 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.682050 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/536941ec-7aff-42ca-b8d9-fba6959b98aa-root\") pod \"node-exporter-785dd\" (UID: \"536941ec-7aff-42ca-b8d9-fba6959b98aa\") " pod="openshift-monitoring/node-exporter-785dd" Jul 06 20:59:59.682644 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.682619 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/536941ec-7aff-42ca-b8d9-fba6959b98aa-node-exporter-accelerators-collector-config\") pod \"node-exporter-785dd\" (UID: \"536941ec-7aff-42ca-b8d9-fba6959b98aa\") " pod="openshift-monitoring/node-exporter-785dd" Jul 06 20:59:59.682775 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.682633 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ab3ede5-c596-4fab-a7db-683fa6c81ebc-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-dhcnh\" (UID: \"0ab3ede5-c596-4fab-a7db-683fa6c81ebc\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dhcnh" Jul 06 20:59:59.683095 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.683075 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/536941ec-7aff-42ca-b8d9-fba6959b98aa-node-exporter-wtmp\") pod \"node-exporter-785dd\" (UID: \"536941ec-7aff-42ca-b8d9-fba6959b98aa\") " pod="openshift-monitoring/node-exporter-785dd" Jul 06 20:59:59.683220 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.683112 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0ab3ede5-c596-4fab-a7db-683fa6c81ebc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-dhcnh\" (UID: \"0ab3ede5-c596-4fab-a7db-683fa6c81ebc\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dhcnh" Jul 06 20:59:59.683623 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.683564 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/536941ec-7aff-42ca-b8d9-fba6959b98aa-metrics-client-ca\") pod \"node-exporter-785dd\" (UID: \"536941ec-7aff-42ca-b8d9-fba6959b98aa\") " pod="openshift-monitoring/node-exporter-785dd" Jul 06 20:59:59.685121 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.685040 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/536941ec-7aff-42ca-b8d9-fba6959b98aa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-785dd\" (UID: \"536941ec-7aff-42ca-b8d9-fba6959b98aa\") " pod="openshift-monitoring/node-exporter-785dd" Jul 06 20:59:59.687518 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.687456 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ab3ede5-c596-4fab-a7db-683fa6c81ebc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-dhcnh\" (UID: \"0ab3ede5-c596-4fab-a7db-683fa6c81ebc\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dhcnh" Jul 06 20:59:59.687670 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.687456 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/536941ec-7aff-42ca-b8d9-fba6959b98aa-node-exporter-tls\") pod \"node-exporter-785dd\" (UID: \"536941ec-7aff-42ca-b8d9-fba6959b98aa\") " pod="openshift-monitoring/node-exporter-785dd" Jul 06 20:59:59.688362 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.688335 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ab3ede5-c596-4fab-a7db-683fa6c81ebc-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-dhcnh\" (UID: \"0ab3ede5-c596-4fab-a7db-683fa6c81ebc\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dhcnh" Jul 06 20:59:59.693481 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.693446 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4fvs\" (UniqueName: \"kubernetes.io/projected/536941ec-7aff-42ca-b8d9-fba6959b98aa-kube-api-access-v4fvs\") pod \"node-exporter-785dd\" (UID: \"536941ec-7aff-42ca-b8d9-fba6959b98aa\") " pod="openshift-monitoring/node-exporter-785dd" Jul 06 20:59:59.693677 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.693647 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfh45\" (UniqueName: \"kubernetes.io/projected/0ab3ede5-c596-4fab-a7db-683fa6c81ebc-kube-api-access-vfh45\") pod \"kube-state-metrics-7c9dbb8b7d-dhcnh\" (UID: \"0ab3ede5-c596-4fab-a7db-683fa6c81ebc\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dhcnh" Jul 06 20:59:59.796360 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.796268 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dhcnh" Jul 06 20:59:59.827435 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.827387 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-785dd" Jul 06 20:59:59.864114 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:59:59.864072 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536941ec_7aff_42ca_b8d9_fba6959b98aa.slice/crio-0226713cb02decbf416daed865e904eefb158db1f72049c5350331143509ea6d WatchSource:0}: Error finding container 0226713cb02decbf416daed865e904eefb158db1f72049c5350331143509ea6d: Status 404 returned error can't find the container with id 0226713cb02decbf416daed865e904eefb158db1f72049c5350331143509ea6d Jul 06 20:59:59.987290 ip-10-0-136-207 kubenswrapper[2577]: I0706 20:59:59.987251 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dhcnh"] Jul 06 20:59:59.990875 ip-10-0-136-207 kubenswrapper[2577]: W0706 20:59:59.990836 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab3ede5_c596_4fab_a7db_683fa6c81ebc.slice/crio-cb9134a5e4c2e64449961fd206b15c456a0aa990762bba34266380da058f39b2 WatchSource:0}: Error finding container cb9134a5e4c2e64449961fd206b15c456a0aa990762bba34266380da058f39b2: Status 404 returned error can't find the container with id cb9134a5e4c2e64449961fd206b15c456a0aa990762bba34266380da058f39b2 Jul 06 21:00:00.575502 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.575452 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dhcnh" event={"ID":"0ab3ede5-c596-4fab-a7db-683fa6c81ebc","Type":"ContainerStarted","Data":"cb9134a5e4c2e64449961fd206b15c456a0aa990762bba34266380da058f39b2"} Jul 06 21:00:00.577143 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.577042 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-785dd" event={"ID":"536941ec-7aff-42ca-b8d9-fba6959b98aa","Type":"ContainerStarted","Data":"0226713cb02decbf416daed865e904eefb158db1f72049c5350331143509ea6d"} Jul 06 21:00:00.601954 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.601916 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 21:00:00.625738 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.623487 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.625738 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.624797 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 21:00:00.647845 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.647804 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-r2cgf\"" Jul 06 21:00:00.648481 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.647861 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 06 21:00:00.648481 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.648220 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 06 21:00:00.648481 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.648408 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 06 21:00:00.648703 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.648591 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 06 21:00:00.648791 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.648775 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 06 21:00:00.649095 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.648996 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 06 21:00:00.653598 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.653564 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 06 21:00:00.669925 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.669888 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 06 21:00:00.690691 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.690506 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0f857be8-1333-4976-8193-892304f05ff3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.690691 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.690557 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0f857be8-1333-4976-8193-892304f05ff3-config-volume\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.690691 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.690590 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0f857be8-1333-4976-8193-892304f05ff3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.690691 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.690620 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f857be8-1333-4976-8193-892304f05ff3-config-out\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.690691 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.690666 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjml6\" (UniqueName: \"kubernetes.io/projected/0f857be8-1333-4976-8193-892304f05ff3-kube-api-access-cjml6\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.691101 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.690702 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f857be8-1333-4976-8193-892304f05ff3-web-config\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.691101 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.690728 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f857be8-1333-4976-8193-892304f05ff3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.691101 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.690967 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f857be8-1333-4976-8193-892304f05ff3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.691101 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.690998 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f857be8-1333-4976-8193-892304f05ff3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.691101 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.691038 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f857be8-1333-4976-8193-892304f05ff3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.691101 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.691064 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f857be8-1333-4976-8193-892304f05ff3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.691101 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.691073 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 06 21:00:00.691101 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.691091 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0f857be8-1333-4976-8193-892304f05ff3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.691398 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.691123 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0f857be8-1333-4976-8193-892304f05ff3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.791502 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.791459 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0f857be8-1333-4976-8193-892304f05ff3-config-volume\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.791674 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.791511 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0f857be8-1333-4976-8193-892304f05ff3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.791674 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.791547 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f857be8-1333-4976-8193-892304f05ff3-config-out\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.791674 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.791588 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjml6\" (UniqueName: \"kubernetes.io/projected/0f857be8-1333-4976-8193-892304f05ff3-kube-api-access-cjml6\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.791674 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.791627 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f857be8-1333-4976-8193-892304f05ff3-web-config\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.791674 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.791657 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f857be8-1333-4976-8193-892304f05ff3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.792637 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.791695 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f857be8-1333-4976-8193-892304f05ff3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.792763 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.792681 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f857be8-1333-4976-8193-892304f05ff3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.792763 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.792711 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f857be8-1333-4976-8193-892304f05ff3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.792763 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.792741 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f857be8-1333-4976-8193-892304f05ff3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.792898 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.792769 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0f857be8-1333-4976-8193-892304f05ff3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.792898 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.792810 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0f857be8-1333-4976-8193-892304f05ff3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.792898 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.792856 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0f857be8-1333-4976-8193-892304f05ff3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.793038 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:00:00.792999 2577 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 06 21:00:00.793102 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:00:00.793089 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f857be8-1333-4976-8193-892304f05ff3-secret-alertmanager-main-tls podName:0f857be8-1333-4976-8193-892304f05ff3 nodeName:}" failed. No retries permitted until 2026-07-06 21:00:01.293065808 +0000 UTC m=+85.611289068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/0f857be8-1333-4976-8193-892304f05ff3-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "0f857be8-1333-4976-8193-892304f05ff3") : secret "alertmanager-main-tls" not found Jul 06 21:00:00.795610 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.795254 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f857be8-1333-4976-8193-892304f05ff3-config-out\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.795610 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.795446 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f857be8-1333-4976-8193-892304f05ff3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.795831 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.795794 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0f857be8-1333-4976-8193-892304f05ff3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.798429 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.797987 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f857be8-1333-4976-8193-892304f05ff3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.798429 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.798382 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0f857be8-1333-4976-8193-892304f05ff3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.801430 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.801272 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0f857be8-1333-4976-8193-892304f05ff3-config-volume\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.801430 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.801381 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f857be8-1333-4976-8193-892304f05ff3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.801639 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.801458 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f857be8-1333-4976-8193-892304f05ff3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.802198 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.802141 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0f857be8-1333-4976-8193-892304f05ff3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.802300 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.802235 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f857be8-1333-4976-8193-892304f05ff3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.804937 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.804800 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f857be8-1333-4976-8193-892304f05ff3-web-config\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:00.816949 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:00.816905 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjml6\" (UniqueName: \"kubernetes.io/projected/0f857be8-1333-4976-8193-892304f05ff3-kube-api-access-cjml6\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:01.297199 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:01.297151 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0f857be8-1333-4976-8193-892304f05ff3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:01.297421 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:00:01.297327 2577 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 06 21:00:01.297421 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:00:01.297399 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f857be8-1333-4976-8193-892304f05ff3-secret-alertmanager-main-tls podName:0f857be8-1333-4976-8193-892304f05ff3 nodeName:}" failed. No retries permitted until 2026-07-06 21:00:02.297380314 +0000 UTC m=+86.615603567 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/0f857be8-1333-4976-8193-892304f05ff3-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "0f857be8-1333-4976-8193-892304f05ff3") : secret "alertmanager-main-tls" not found Jul 06 21:00:02.305344 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:02.305288 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0f857be8-1333-4976-8193-892304f05ff3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:02.311047 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:02.309442 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0f857be8-1333-4976-8193-892304f05ff3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0f857be8-1333-4976-8193-892304f05ff3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:02.441365 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:02.441323 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 21:00:02.591587 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:02.590528 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dhcnh" event={"ID":"0ab3ede5-c596-4fab-a7db-683fa6c81ebc","Type":"ContainerStarted","Data":"554fafc3afa2cbe7a59503a50d5fa6db6ed3a33215cc7854b5196babcc209032"} Jul 06 21:00:02.591587 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:02.590635 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dhcnh" event={"ID":"0ab3ede5-c596-4fab-a7db-683fa6c81ebc","Type":"ContainerStarted","Data":"bfd5d9881ad506a73d91504802dac13c6ad0118942a5bf6c92488b310efbb506"} Jul 06 21:00:02.591587 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:02.590668 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dhcnh" event={"ID":"0ab3ede5-c596-4fab-a7db-683fa6c81ebc","Type":"ContainerStarted","Data":"dc14237f92c2403ee065224de1b1b4151aa0ff1014ece5d93c116982be977f44"} Jul 06 21:00:02.593912 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:02.593880 2577 generic.go:356] "Generic (PLEG): container finished" podID="536941ec-7aff-42ca-b8d9-fba6959b98aa" containerID="dff2374012e22c8f2aa40122d165b053cf16dca03c7d56e103bb267f6882f9d8" exitCode=0 Jul 06 21:00:02.598851 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:02.594695 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-785dd" event={"ID":"536941ec-7aff-42ca-b8d9-fba6959b98aa","Type":"ContainerDied","Data":"dff2374012e22c8f2aa40122d165b053cf16dca03c7d56e103bb267f6882f9d8"} Jul 06 21:00:02.626539 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:02.626143 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-dhcnh" podStartSLOduration=1.55271486 podStartE2EDuration="3.626120877s" totalImagesPullingTime="2.073406017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 20:59:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:00:02.625059367 +0000 UTC m=+86.943282654" watchObservedRunningTime="2026-07-06 21:00:02.626120877 +0000 UTC m=+86.944344138" Jul 06 21:00:02.707586 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:02.707476 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 21:00:02.715240 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:00:02.714220 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f857be8_1333_4976_8193_892304f05ff3.slice/crio-22196fb012420eed08f58249dea9109d785a4479b55698ebdce39b35d00d600c WatchSource:0}: Error finding container 22196fb012420eed08f58249dea9109d785a4479b55698ebdce39b35d00d600c: Status 404 returned error can't find the container with id 22196fb012420eed08f58249dea9109d785a4479b55698ebdce39b35d00d600c Jul 06 21:00:03.607900 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:03.607797 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-785dd" event={"ID":"536941ec-7aff-42ca-b8d9-fba6959b98aa","Type":"ContainerStarted","Data":"742b209b5f2b9138d59be1b6ebc9be93271150a4b7ef21228eefa181c569a301"} Jul 06 21:00:03.608467 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:03.608430 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-785dd" event={"ID":"536941ec-7aff-42ca-b8d9-fba6959b98aa","Type":"ContainerStarted","Data":"ba34233fedaa99f95e23aa936d371dc1151243653d975816293bad51c3b8b944"} Jul 06 21:00:03.610547 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:03.610508 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0f857be8-1333-4976-8193-892304f05ff3","Type":"ContainerStarted","Data":"22196fb012420eed08f58249dea9109d785a4479b55698ebdce39b35d00d600c"} Jul 06 21:00:03.635650 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:03.635501 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-785dd" podStartSLOduration=2.317685536 podStartE2EDuration="4.635473859s" totalImagesPullingTime="2.197482988s" totalInitContainerRuntime="120.305335ms" isStatefulPod=true podCreationTimestamp="2026-07-06 20:59:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:00:03.634569889 +0000 UTC m=+87.952793151" watchObservedRunningTime="2026-07-06 21:00:03.635473859 +0000 UTC m=+87.953697118" Jul 06 21:00:03.766185 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:03.766126 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-645d5b8597-84426" podUID="ad03b367-ccbf-4b95-99f4-29e492b386ac" containerName="registry" containerID="cri-o://61281e7e49957bdcf223f919fdc02d5eacbd8147fd15e577b7b38a4452958c29" gracePeriod=30 Jul 06 21:00:04.279380 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.279340 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-r2hp9"] Jul 06 21:00:04.298073 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.297965 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-r2hp9"] Jul 06 21:00:04.298245 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.298149 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-r2hp9" Jul 06 21:00:04.301505 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.301474 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-qhknj\"" Jul 06 21:00:04.302532 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.302508 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 06 21:00:04.323734 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.323691 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/374e460c-d8aa-4a91-972b-2cf27ff28dc2-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-r2hp9\" (UID: \"374e460c-d8aa-4a91-972b-2cf27ff28dc2\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-r2hp9" Jul 06 21:00:04.323926 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.323754 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtcvd\" (UniqueName: \"kubernetes.io/projected/374e460c-d8aa-4a91-972b-2cf27ff28dc2-kube-api-access-gtcvd\") pod \"monitoring-plugin-d5dcff7cf-r2hp9\" (UID: \"374e460c-d8aa-4a91-972b-2cf27ff28dc2\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-r2hp9" Jul 06 21:00:04.402109 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.397471 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 21:00:04.425460 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.424223 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ad03b367-ccbf-4b95-99f4-29e492b386ac-image-registry-private-configuration\") pod \"ad03b367-ccbf-4b95-99f4-29e492b386ac\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " Jul 06 21:00:04.425460 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.424293 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ad03b367-ccbf-4b95-99f4-29e492b386ac-installation-pull-secrets\") pod \"ad03b367-ccbf-4b95-99f4-29e492b386ac\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " Jul 06 21:00:04.425460 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.424337 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ad03b367-ccbf-4b95-99f4-29e492b386ac-trusted-ca\") pod \"ad03b367-ccbf-4b95-99f4-29e492b386ac\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " Jul 06 21:00:04.425460 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.424363 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ad03b367-ccbf-4b95-99f4-29e492b386ac-registry-certificates\") pod \"ad03b367-ccbf-4b95-99f4-29e492b386ac\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " Jul 06 21:00:04.425460 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.424392 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sx9c7\" (UniqueName: \"kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-kube-api-access-sx9c7\") pod \"ad03b367-ccbf-4b95-99f4-29e492b386ac\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " Jul 06 21:00:04.425460 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.424426 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ad03b367-ccbf-4b95-99f4-29e492b386ac-ca-trust-extracted\") pod \"ad03b367-ccbf-4b95-99f4-29e492b386ac\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " Jul 06 21:00:04.425460 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.424453 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-bound-sa-token\") pod \"ad03b367-ccbf-4b95-99f4-29e492b386ac\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " Jul 06 21:00:04.425460 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.424496 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-registry-tls\") pod \"ad03b367-ccbf-4b95-99f4-29e492b386ac\" (UID: \"ad03b367-ccbf-4b95-99f4-29e492b386ac\") " Jul 06 21:00:04.425460 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.424614 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtcvd\" (UniqueName: \"kubernetes.io/projected/374e460c-d8aa-4a91-972b-2cf27ff28dc2-kube-api-access-gtcvd\") pod \"monitoring-plugin-d5dcff7cf-r2hp9\" (UID: \"374e460c-d8aa-4a91-972b-2cf27ff28dc2\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-r2hp9" Jul 06 21:00:04.425460 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.424749 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/374e460c-d8aa-4a91-972b-2cf27ff28dc2-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-r2hp9\" (UID: \"374e460c-d8aa-4a91-972b-2cf27ff28dc2\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-r2hp9" Jul 06 21:00:04.425460 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.424803 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad03b367-ccbf-4b95-99f4-29e492b386ac-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ad03b367-ccbf-4b95-99f4-29e492b386ac" (UID: "ad03b367-ccbf-4b95-99f4-29e492b386ac"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:00:04.425460 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:00:04.424874 2577 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 06 21:00:04.425460 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:00:04.424959 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/374e460c-d8aa-4a91-972b-2cf27ff28dc2-monitoring-plugin-cert podName:374e460c-d8aa-4a91-972b-2cf27ff28dc2 nodeName:}" failed. No retries permitted until 2026-07-06 21:00:04.924935092 +0000 UTC m=+89.243158341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/374e460c-d8aa-4a91-972b-2cf27ff28dc2-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-r2hp9" (UID: "374e460c-d8aa-4a91-972b-2cf27ff28dc2") : secret "monitoring-plugin-cert" not found Jul 06 21:00:04.426273 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.425968 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad03b367-ccbf-4b95-99f4-29e492b386ac-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ad03b367-ccbf-4b95-99f4-29e492b386ac" (UID: "ad03b367-ccbf-4b95-99f4-29e492b386ac"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:00:04.432177 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.432108 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad03b367-ccbf-4b95-99f4-29e492b386ac-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ad03b367-ccbf-4b95-99f4-29e492b386ac" (UID: "ad03b367-ccbf-4b95-99f4-29e492b386ac"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:00:04.432177 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.432155 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ad03b367-ccbf-4b95-99f4-29e492b386ac" (UID: "ad03b367-ccbf-4b95-99f4-29e492b386ac"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:00:04.433472 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.433225 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad03b367-ccbf-4b95-99f4-29e492b386ac-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ad03b367-ccbf-4b95-99f4-29e492b386ac" (UID: "ad03b367-ccbf-4b95-99f4-29e492b386ac"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:00:04.433472 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.433306 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-kube-api-access-sx9c7" (OuterVolumeSpecName: "kube-api-access-sx9c7") pod "ad03b367-ccbf-4b95-99f4-29e492b386ac" (UID: "ad03b367-ccbf-4b95-99f4-29e492b386ac"). InnerVolumeSpecName "kube-api-access-sx9c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:00:04.437857 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.435974 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ad03b367-ccbf-4b95-99f4-29e492b386ac" (UID: "ad03b367-ccbf-4b95-99f4-29e492b386ac"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:00:04.437857 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.436327 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad03b367-ccbf-4b95-99f4-29e492b386ac-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ad03b367-ccbf-4b95-99f4-29e492b386ac" (UID: "ad03b367-ccbf-4b95-99f4-29e492b386ac"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:00:04.456205 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.456164 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtcvd\" (UniqueName: \"kubernetes.io/projected/374e460c-d8aa-4a91-972b-2cf27ff28dc2-kube-api-access-gtcvd\") pod \"monitoring-plugin-d5dcff7cf-r2hp9\" (UID: \"374e460c-d8aa-4a91-972b-2cf27ff28dc2\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-r2hp9" Jul 06 21:00:04.525592 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.525547 2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ad03b367-ccbf-4b95-99f4-29e492b386ac-image-registry-private-configuration\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:00:04.525592 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.525589 2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ad03b367-ccbf-4b95-99f4-29e492b386ac-installation-pull-secrets\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:00:04.525592 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.525603 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ad03b367-ccbf-4b95-99f4-29e492b386ac-trusted-ca\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:00:04.525889 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.525614 2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ad03b367-ccbf-4b95-99f4-29e492b386ac-registry-certificates\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:00:04.525889 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.525625 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx9c7\" (UniqueName: \"kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-kube-api-access-sx9c7\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:00:04.525889 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.525634 2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ad03b367-ccbf-4b95-99f4-29e492b386ac-ca-trust-extracted\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:00:04.525889 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.525648 2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-bound-sa-token\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:00:04.525889 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.525663 2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ad03b367-ccbf-4b95-99f4-29e492b386ac-registry-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:00:04.621548 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.621514 2577 generic.go:356] "Generic (PLEG): container finished" podID="ad03b367-ccbf-4b95-99f4-29e492b386ac" containerID="61281e7e49957bdcf223f919fdc02d5eacbd8147fd15e577b7b38a4452958c29" exitCode=0 Jul 06 21:00:04.622039 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.621663 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-645d5b8597-84426" Jul 06 21:00:04.622111 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.622074 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-645d5b8597-84426" event={"ID":"ad03b367-ccbf-4b95-99f4-29e492b386ac","Type":"ContainerDied","Data":"61281e7e49957bdcf223f919fdc02d5eacbd8147fd15e577b7b38a4452958c29"} Jul 06 21:00:04.622166 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.622136 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-645d5b8597-84426" event={"ID":"ad03b367-ccbf-4b95-99f4-29e492b386ac","Type":"ContainerDied","Data":"feb9fa94368634654bc9db29bec694b09569ab0b9e86891cb1fb970c393231e9"} Jul 06 21:00:04.622222 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.622187 2577 scope.go:117] "RemoveContainer" containerID="61281e7e49957bdcf223f919fdc02d5eacbd8147fd15e577b7b38a4452958c29" Jul 06 21:00:04.624522 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.624486 2577 generic.go:356] "Generic (PLEG): container finished" podID="0f857be8-1333-4976-8193-892304f05ff3" containerID="e2a3a5819c89931833889b1fb52116a9d49523a17019c2ab8e9e357d1b66d47a" exitCode=0 Jul 06 21:00:04.626147 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.626113 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0f857be8-1333-4976-8193-892304f05ff3","Type":"ContainerDied","Data":"e2a3a5819c89931833889b1fb52116a9d49523a17019c2ab8e9e357d1b66d47a"} Jul 06 21:00:04.635189 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.635162 2577 scope.go:117] "RemoveContainer" containerID="61281e7e49957bdcf223f919fdc02d5eacbd8147fd15e577b7b38a4452958c29" Jul 06 21:00:04.635612 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:00:04.635584 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61281e7e49957bdcf223f919fdc02d5eacbd8147fd15e577b7b38a4452958c29\": container with ID starting with 61281e7e49957bdcf223f919fdc02d5eacbd8147fd15e577b7b38a4452958c29 not found: ID does not exist" containerID="61281e7e49957bdcf223f919fdc02d5eacbd8147fd15e577b7b38a4452958c29" Jul 06 21:00:04.635715 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.635629 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61281e7e49957bdcf223f919fdc02d5eacbd8147fd15e577b7b38a4452958c29"} err="failed to get container status \"61281e7e49957bdcf223f919fdc02d5eacbd8147fd15e577b7b38a4452958c29\": rpc error: code = NotFound desc = could not find container \"61281e7e49957bdcf223f919fdc02d5eacbd8147fd15e577b7b38a4452958c29\": container with ID starting with 61281e7e49957bdcf223f919fdc02d5eacbd8147fd15e577b7b38a4452958c29 not found: ID does not exist" Jul 06 21:00:04.684210 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.684167 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-645d5b8597-84426"] Jul 06 21:00:04.690344 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.690295 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-645d5b8597-84426"] Jul 06 21:00:04.929399 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:04.929349 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/374e460c-d8aa-4a91-972b-2cf27ff28dc2-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-r2hp9\" (UID: \"374e460c-d8aa-4a91-972b-2cf27ff28dc2\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-r2hp9" Jul 06 21:00:04.929606 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:00:04.929544 2577 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 06 21:00:04.929666 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:00:04.929634 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/374e460c-d8aa-4a91-972b-2cf27ff28dc2-monitoring-plugin-cert podName:374e460c-d8aa-4a91-972b-2cf27ff28dc2 nodeName:}" failed. No retries permitted until 2026-07-06 21:00:05.9296108 +0000 UTC m=+90.247834065 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/374e460c-d8aa-4a91-972b-2cf27ff28dc2-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-r2hp9" (UID: "374e460c-d8aa-4a91-972b-2cf27ff28dc2") : secret "monitoring-plugin-cert" not found Jul 06 21:00:05.906026 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.903854 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 21:00:05.906026 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.904303 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad03b367-ccbf-4b95-99f4-29e492b386ac" containerName="registry" Jul 06 21:00:05.906026 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.904324 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad03b367-ccbf-4b95-99f4-29e492b386ac" containerName="registry" Jul 06 21:00:05.906026 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.904407 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad03b367-ccbf-4b95-99f4-29e492b386ac" containerName="registry" Jul 06 21:00:05.925703 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.925665 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:05.934318 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.933100 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7jsoitjkrje0t\"" Jul 06 21:00:05.934318 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.933545 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 21:00:05.934318 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.933953 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 21:00:05.934318 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.934269 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 21:00:05.934975 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.934748 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xstf2\"" Jul 06 21:00:05.934975 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.934939 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 21:00:05.935215 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.935148 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 21:00:05.937315 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.935358 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 21:00:05.937315 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.935595 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 21:00:05.937315 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.935782 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 21:00:05.937315 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.935896 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 21:00:05.937315 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.936123 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 21:00:05.937315 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.936310 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 21:00:05.937315 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.936500 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 21:00:05.937315 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.936764 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 21:00:05.938553 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.938041 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 21:00:05.938553 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.938463 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 21:00:05.945046 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.939531 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80b3f415-31f0-400d-a51b-b905f18fd8a3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:05.945046 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.939579 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80b3f415-31f0-400d-a51b-b905f18fd8a3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:05.945046 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.940087 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/80b3f415-31f0-400d-a51b-b905f18fd8a3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:05.945046 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.940122 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/80b3f415-31f0-400d-a51b-b905f18fd8a3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:05.945046 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.940153 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80b3f415-31f0-400d-a51b-b905f18fd8a3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:05.945046 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.940224 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/80b3f415-31f0-400d-a51b-b905f18fd8a3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:05.945046 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.940259 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/80b3f415-31f0-400d-a51b-b905f18fd8a3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:05.945046 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.940288 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80b3f415-31f0-400d-a51b-b905f18fd8a3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:05.945046 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.940318 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/80b3f415-31f0-400d-a51b-b905f18fd8a3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:05.945046 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.940343 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80b3f415-31f0-400d-a51b-b905f18fd8a3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:05.945046 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.940383 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/80b3f415-31f0-400d-a51b-b905f18fd8a3-config-out\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:05.945046 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.940456 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/374e460c-d8aa-4a91-972b-2cf27ff28dc2-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-r2hp9\" (UID: \"374e460c-d8aa-4a91-972b-2cf27ff28dc2\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-r2hp9" Jul 06 21:00:05.945046 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.942483 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/80b3f415-31f0-400d-a51b-b905f18fd8a3-config\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:05.945796 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.942542 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/80b3f415-31f0-400d-a51b-b905f18fd8a3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:05.945796 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.942579 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80b3f415-31f0-400d-a51b-b905f18fd8a3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:05.945796 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.942620 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/80b3f415-31f0-400d-a51b-b905f18fd8a3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:05.945796 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.942655 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7tnj\" (UniqueName: \"kubernetes.io/projected/80b3f415-31f0-400d-a51b-b905f18fd8a3-kube-api-access-h7tnj\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:05.945796 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.942688 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/80b3f415-31f0-400d-a51b-b905f18fd8a3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:05.945796 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.942718 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/80b3f415-31f0-400d-a51b-b905f18fd8a3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:05.945796 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.942739 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/80b3f415-31f0-400d-a51b-b905f18fd8a3-web-config\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:05.945796 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.942772 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/80b3f415-31f0-400d-a51b-b905f18fd8a3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:05.945796 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.943836 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/374e460c-d8aa-4a91-972b-2cf27ff28dc2-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-r2hp9\" (UID: \"374e460c-d8aa-4a91-972b-2cf27ff28dc2\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-r2hp9" Jul 06 21:00:05.954640 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:05.954595 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 21:00:06.043786 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.043743 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/80b3f415-31f0-400d-a51b-b905f18fd8a3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.043979 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.043793 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/80b3f415-31f0-400d-a51b-b905f18fd8a3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.043979 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.043831 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80b3f415-31f0-400d-a51b-b905f18fd8a3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.043979 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.043855 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/80b3f415-31f0-400d-a51b-b905f18fd8a3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.043979 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.043873 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80b3f415-31f0-400d-a51b-b905f18fd8a3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.043979 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.043908 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/80b3f415-31f0-400d-a51b-b905f18fd8a3-config-out\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.043979 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.043968 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/80b3f415-31f0-400d-a51b-b905f18fd8a3-config\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.044399 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.043998 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/80b3f415-31f0-400d-a51b-b905f18fd8a3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.044399 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.044054 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80b3f415-31f0-400d-a51b-b905f18fd8a3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.044399 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.044094 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/80b3f415-31f0-400d-a51b-b905f18fd8a3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.044399 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.044124 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7tnj\" (UniqueName: \"kubernetes.io/projected/80b3f415-31f0-400d-a51b-b905f18fd8a3-kube-api-access-h7tnj\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.044399 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.044153 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/80b3f415-31f0-400d-a51b-b905f18fd8a3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.044399 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.044184 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/80b3f415-31f0-400d-a51b-b905f18fd8a3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.044399 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.044206 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/80b3f415-31f0-400d-a51b-b905f18fd8a3-web-config\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.044399 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.044246 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/80b3f415-31f0-400d-a51b-b905f18fd8a3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.044399 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.044282 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80b3f415-31f0-400d-a51b-b905f18fd8a3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.044399 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.044311 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80b3f415-31f0-400d-a51b-b905f18fd8a3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.044399 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.044343 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/80b3f415-31f0-400d-a51b-b905f18fd8a3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.044399 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.044373 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/80b3f415-31f0-400d-a51b-b905f18fd8a3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.044399 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.044401 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80b3f415-31f0-400d-a51b-b905f18fd8a3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.047928 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.047351 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/80b3f415-31f0-400d-a51b-b905f18fd8a3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.047928 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.047666 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80b3f415-31f0-400d-a51b-b905f18fd8a3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.047928 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.047715 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/80b3f415-31f0-400d-a51b-b905f18fd8a3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.049505 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.048909 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80b3f415-31f0-400d-a51b-b905f18fd8a3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.050981 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.048938 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/80b3f415-31f0-400d-a51b-b905f18fd8a3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.051134 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.049050 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80b3f415-31f0-400d-a51b-b905f18fd8a3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.051134 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.049338 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/80b3f415-31f0-400d-a51b-b905f18fd8a3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.051134 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.049527 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80b3f415-31f0-400d-a51b-b905f18fd8a3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.051134 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.050361 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80b3f415-31f0-400d-a51b-b905f18fd8a3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.051134 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.050664 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/80b3f415-31f0-400d-a51b-b905f18fd8a3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.052247 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.052224 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/80b3f415-31f0-400d-a51b-b905f18fd8a3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.053978 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.053943 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80b3f415-31f0-400d-a51b-b905f18fd8a3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.054731 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.054468 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/80b3f415-31f0-400d-a51b-b905f18fd8a3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.054875 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.054733 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/80b3f415-31f0-400d-a51b-b905f18fd8a3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.054875 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.054749 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/80b3f415-31f0-400d-a51b-b905f18fd8a3-config\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.055962 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.055731 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/80b3f415-31f0-400d-a51b-b905f18fd8a3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.055962 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.055923 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/80b3f415-31f0-400d-a51b-b905f18fd8a3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.056146 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.055997 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/80b3f415-31f0-400d-a51b-b905f18fd8a3-config-out\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.057191 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.056956 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/80b3f415-31f0-400d-a51b-b905f18fd8a3-web-config\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.058354 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.058298 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7tnj\" (UniqueName: \"kubernetes.io/projected/80b3f415-31f0-400d-a51b-b905f18fd8a3-kube-api-access-h7tnj\") pod \"prometheus-k8s-0\" (UID: \"80b3f415-31f0-400d-a51b-b905f18fd8a3\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.111112 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.111058 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-r2hp9" Jul 06 21:00:06.176720 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.176622 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad03b367-ccbf-4b95-99f4-29e492b386ac" path="/var/lib/kubelet/pods/ad03b367-ccbf-4b95-99f4-29e492b386ac/volumes" Jul 06 21:00:06.246316 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.246275 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:06.344903 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.344833 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-r2hp9"] Jul 06 21:00:06.349278 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:00:06.349243 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod374e460c_d8aa_4a91_972b_2cf27ff28dc2.slice/crio-b205ca5bd181a777fc052836be9a8a0261e77c61fd9e93b9bf98c6d781b93149 WatchSource:0}: Error finding container b205ca5bd181a777fc052836be9a8a0261e77c61fd9e93b9bf98c6d781b93149: Status 404 returned error can't find the container with id b205ca5bd181a777fc052836be9a8a0261e77c61fd9e93b9bf98c6d781b93149 Jul 06 21:00:06.477048 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.476986 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 21:00:06.479372 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:00:06.479330 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b3f415_31f0_400d_a51b_b905f18fd8a3.slice/crio-90fbe07362a636521c970728a2726d1d529f8894b40f2b68ad314e30cdc111a0 WatchSource:0}: Error finding container 90fbe07362a636521c970728a2726d1d529f8894b40f2b68ad314e30cdc111a0: Status 404 returned error can't find the container with id 90fbe07362a636521c970728a2726d1d529f8894b40f2b68ad314e30cdc111a0 Jul 06 21:00:06.637242 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.637194 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80b3f415-31f0-400d-a51b-b905f18fd8a3","Type":"ContainerStarted","Data":"7f0fb9a6573a4235bda3971a4772e7b67518d47bc763d8136b7faf31f08d2046"} Jul 06 21:00:06.637426 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.637293 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80b3f415-31f0-400d-a51b-b905f18fd8a3","Type":"ContainerStarted","Data":"90fbe07362a636521c970728a2726d1d529f8894b40f2b68ad314e30cdc111a0"} Jul 06 21:00:06.639159 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:06.639115 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-r2hp9" event={"ID":"374e460c-d8aa-4a91-972b-2cf27ff28dc2","Type":"ContainerStarted","Data":"b205ca5bd181a777fc052836be9a8a0261e77c61fd9e93b9bf98c6d781b93149"} Jul 06 21:00:07.645391 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:07.645358 2577 generic.go:356] "Generic (PLEG): container finished" podID="80b3f415-31f0-400d-a51b-b905f18fd8a3" containerID="7f0fb9a6573a4235bda3971a4772e7b67518d47bc763d8136b7faf31f08d2046" exitCode=0 Jul 06 21:00:07.645841 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:07.645447 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80b3f415-31f0-400d-a51b-b905f18fd8a3","Type":"ContainerDied","Data":"7f0fb9a6573a4235bda3971a4772e7b67518d47bc763d8136b7faf31f08d2046"} Jul 06 21:00:08.653175 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:08.653140 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0f857be8-1333-4976-8193-892304f05ff3","Type":"ContainerStarted","Data":"d2b29170d462c331db9ffc66a60c9d57868e4b7fa576958b398e7c9db8e02ed8"} Jul 06 21:00:08.653536 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:08.653207 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0f857be8-1333-4976-8193-892304f05ff3","Type":"ContainerStarted","Data":"9d967c2d0ac798ac72de4ea08c11c2e802c6e1b755bf324b6309ed75b656eb7e"} Jul 06 21:00:08.653536 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:08.653233 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0f857be8-1333-4976-8193-892304f05ff3","Type":"ContainerStarted","Data":"a288f7942ac5bbfb91ea039578a940c6655fd68d99f2b84a38528048db54979e"} Jul 06 21:00:08.653536 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:08.653266 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0f857be8-1333-4976-8193-892304f05ff3","Type":"ContainerStarted","Data":"3093dffc5bc57c8c415349e93d54129ffc1f543ce3cbf13d3a15614e53994f37"} Jul 06 21:00:09.660896 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:09.660852 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0f857be8-1333-4976-8193-892304f05ff3","Type":"ContainerStarted","Data":"f777cf3eee08c03648a75b263320c56ba36d3f6d80d247daaaa30e356ca08a86"} Jul 06 21:00:09.662536 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:09.662473 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-r2hp9" event={"ID":"374e460c-d8aa-4a91-972b-2cf27ff28dc2","Type":"ContainerStarted","Data":"b4d781335fa7fc7042107d37e9ba6b7d9fb6488459b5d9a3e6c257686bc08ec4"} Jul 06 21:00:09.662756 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:09.662719 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-r2hp9" Jul 06 21:00:09.668154 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:09.668126 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-r2hp9" Jul 06 21:00:09.684810 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:09.684696 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-r2hp9" podStartSLOduration=3.014971848 podStartE2EDuration="5.684677305s" totalImagesPullingTime="2.669705457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:00:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:00:09.682620512 +0000 UTC m=+94.000843772" watchObservedRunningTime="2026-07-06 21:00:09.684677305 +0000 UTC m=+94.002900566" Jul 06 21:00:10.674247 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:10.674184 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0f857be8-1333-4976-8193-892304f05ff3","Type":"ContainerStarted","Data":"6197c4f5cac3f38ecfac34560b007078a927c8db18795a13a0982925c03b45e0"} Jul 06 21:00:10.709119 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:10.709059 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.797064952 podStartE2EDuration="10.709041586s" totalImagesPullingTime="5.885741037s" totalInitContainerRuntime="26.235597ms" isStatefulPod=false podCreationTimestamp="2026-07-06 21:00:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:00:10.707707498 +0000 UTC m=+95.025930786" watchObservedRunningTime="2026-07-06 21:00:10.709041586 +0000 UTC m=+95.027264845" Jul 06 21:00:12.683066 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:12.683003 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80b3f415-31f0-400d-a51b-b905f18fd8a3","Type":"ContainerStarted","Data":"6e59e56d337daa3f76b2ef702059a16afe5ae4ab668dd157cc862e1e43d15d11"} Jul 06 21:00:12.683513 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:12.683098 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80b3f415-31f0-400d-a51b-b905f18fd8a3","Type":"ContainerStarted","Data":"21c9c055e995e4446a6ca0f205714e0280193a001ba328fd0c591487cd584f27"} Jul 06 21:00:15.698904 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:15.698860 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80b3f415-31f0-400d-a51b-b905f18fd8a3","Type":"ContainerStarted","Data":"aee747bbe28d449a676ff626cc263c0029d149dd28b1dd9c73bb19d7848bbe65"} Jul 06 21:00:15.699464 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:15.698967 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80b3f415-31f0-400d-a51b-b905f18fd8a3","Type":"ContainerStarted","Data":"262a35aca7d3774151645e7576ee86cd1004ebbfd8d9ab1daa57ea2e1c434d99"} Jul 06 21:00:15.699464 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:15.698996 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80b3f415-31f0-400d-a51b-b905f18fd8a3","Type":"ContainerStarted","Data":"bd313666be575616704cbe5edb4ede982092151f1dd3c6a1d24303aeee95d2f8"} Jul 06 21:00:15.699464 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:15.699055 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80b3f415-31f0-400d-a51b-b905f18fd8a3","Type":"ContainerStarted","Data":"e614559e6a70f943a606443ae974cd507b7355e6611224c457a3749f0d4b4d5d"} Jul 06 21:00:15.737627 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:15.737433 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.680576526 podStartE2EDuration="10.737407706s" totalImagesPullingTime="7.013539812s" totalInitContainerRuntime="43.291368ms" isStatefulPod=false podCreationTimestamp="2026-07-06 21:00:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:00:15.734125129 +0000 UTC m=+100.052348390" watchObservedRunningTime="2026-07-06 21:00:15.737407706 +0000 UTC m=+100.055630967" Jul 06 21:00:16.247414 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:16.247370 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:00:19.531179 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:00:19.531134 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-n8x9c" Jul 06 21:01:06.246866 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:01:06.246826 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:01:06.267520 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:01:06.267493 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:01:06.915704 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:01:06.915672 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 21:03:35.270594 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:35.270555 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gv6zb"] Jul 06 21:03:35.272916 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:35.272895 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gv6zb" Jul 06 21:03:35.275462 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:35.275434 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-mxxqh\"" Jul 06 21:03:35.275587 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:35.275434 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 06 21:03:35.276450 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:35.276429 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 06 21:03:35.284811 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:35.284780 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gv6zb"] Jul 06 21:03:35.333993 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:35.333950 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwrx4\" (UniqueName: \"kubernetes.io/projected/ac84b370-4a2a-4a0d-809c-a1ff18e8c29e-kube-api-access-hwrx4\") pod \"cert-manager-cainjector-559cdd8c87-gv6zb\" (UID: \"ac84b370-4a2a-4a0d-809c-a1ff18e8c29e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gv6zb" Jul 06 21:03:35.334192 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:35.334105 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac84b370-4a2a-4a0d-809c-a1ff18e8c29e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gv6zb\" (UID: \"ac84b370-4a2a-4a0d-809c-a1ff18e8c29e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gv6zb" Jul 06 21:03:35.435139 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:35.435108 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac84b370-4a2a-4a0d-809c-a1ff18e8c29e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gv6zb\" (UID: \"ac84b370-4a2a-4a0d-809c-a1ff18e8c29e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gv6zb" Jul 06 21:03:35.435294 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:35.435169 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwrx4\" (UniqueName: \"kubernetes.io/projected/ac84b370-4a2a-4a0d-809c-a1ff18e8c29e-kube-api-access-hwrx4\") pod \"cert-manager-cainjector-559cdd8c87-gv6zb\" (UID: \"ac84b370-4a2a-4a0d-809c-a1ff18e8c29e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gv6zb" Jul 06 21:03:35.446336 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:35.446295 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac84b370-4a2a-4a0d-809c-a1ff18e8c29e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gv6zb\" (UID: \"ac84b370-4a2a-4a0d-809c-a1ff18e8c29e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gv6zb" Jul 06 21:03:35.446463 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:35.446375 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwrx4\" (UniqueName: \"kubernetes.io/projected/ac84b370-4a2a-4a0d-809c-a1ff18e8c29e-kube-api-access-hwrx4\") pod \"cert-manager-cainjector-559cdd8c87-gv6zb\" (UID: \"ac84b370-4a2a-4a0d-809c-a1ff18e8c29e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gv6zb" Jul 06 21:03:35.593671 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:35.593580 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gv6zb" Jul 06 21:03:35.739820 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:35.739795 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gv6zb"] Jul 06 21:03:35.741672 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:03:35.741648 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac84b370_4a2a_4a0d_809c_a1ff18e8c29e.slice/crio-e23b40eaa61f63423a41cefd56e0e1dc76d5fa1c0ec147f221361ccf80bef521 WatchSource:0}: Error finding container e23b40eaa61f63423a41cefd56e0e1dc76d5fa1c0ec147f221361ccf80bef521: Status 404 returned error can't find the container with id e23b40eaa61f63423a41cefd56e0e1dc76d5fa1c0ec147f221361ccf80bef521 Jul 06 21:03:36.067507 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:36.067472 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qjs2n_64b45669-ed56-41b9-a3ed-1fe5c13508b6/ovn-acl-logging/0.log" Jul 06 21:03:36.068806 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:36.068786 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qjs2n_64b45669-ed56-41b9-a3ed-1fe5c13508b6/ovn-acl-logging/0.log" Jul 06 21:03:36.070862 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:36.070841 2577 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 21:03:36.468814 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:36.468772 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gv6zb" event={"ID":"ac84b370-4a2a-4a0d-809c-a1ff18e8c29e","Type":"ContainerStarted","Data":"e23b40eaa61f63423a41cefd56e0e1dc76d5fa1c0ec147f221361ccf80bef521"} Jul 06 21:03:39.483803 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:39.483760 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gv6zb" event={"ID":"ac84b370-4a2a-4a0d-809c-a1ff18e8c29e","Type":"ContainerStarted","Data":"dc441b904e95ac52786d424317cdf6be60f84279db20dcc0f5642de10b1d29c5"} Jul 06 21:03:39.504483 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:39.504425 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gv6zb" podStartSLOduration=1.458692364 podStartE2EDuration="4.504407712s" totalImagesPullingTime="3.045715348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:03:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:03:39.502322253 +0000 UTC m=+303.820545513" watchObservedRunningTime="2026-07-06 21:03:39.504407712 +0000 UTC m=+303.822630972" Jul 06 21:03:43.916241 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:43.916197 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-zsb6m"] Jul 06 21:03:43.919068 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:43.919039 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-zsb6m" Jul 06 21:03:43.923891 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:43.923864 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-wgccm\"" Jul 06 21:03:43.932571 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:43.932539 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-zsb6m"] Jul 06 21:03:44.011093 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:44.011054 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4cb5dce-28ff-4fb3-833d-68bc06689385-bound-sa-token\") pod \"cert-manager-55b4d954d5-zsb6m\" (UID: \"a4cb5dce-28ff-4fb3-833d-68bc06689385\") " pod="cert-manager/cert-manager-55b4d954d5-zsb6m" Jul 06 21:03:44.011278 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:44.011142 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb66t\" (UniqueName: \"kubernetes.io/projected/a4cb5dce-28ff-4fb3-833d-68bc06689385-kube-api-access-cb66t\") pod \"cert-manager-55b4d954d5-zsb6m\" (UID: \"a4cb5dce-28ff-4fb3-833d-68bc06689385\") " pod="cert-manager/cert-manager-55b4d954d5-zsb6m" Jul 06 21:03:44.111963 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:44.111923 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4cb5dce-28ff-4fb3-833d-68bc06689385-bound-sa-token\") pod \"cert-manager-55b4d954d5-zsb6m\" (UID: \"a4cb5dce-28ff-4fb3-833d-68bc06689385\") " pod="cert-manager/cert-manager-55b4d954d5-zsb6m" Jul 06 21:03:44.111963 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:44.111971 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb66t\" (UniqueName: \"kubernetes.io/projected/a4cb5dce-28ff-4fb3-833d-68bc06689385-kube-api-access-cb66t\") pod \"cert-manager-55b4d954d5-zsb6m\" (UID: \"a4cb5dce-28ff-4fb3-833d-68bc06689385\") " pod="cert-manager/cert-manager-55b4d954d5-zsb6m" Jul 06 21:03:44.121212 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:44.121183 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4cb5dce-28ff-4fb3-833d-68bc06689385-bound-sa-token\") pod \"cert-manager-55b4d954d5-zsb6m\" (UID: \"a4cb5dce-28ff-4fb3-833d-68bc06689385\") " pod="cert-manager/cert-manager-55b4d954d5-zsb6m" Jul 06 21:03:44.121364 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:44.121239 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb66t\" (UniqueName: \"kubernetes.io/projected/a4cb5dce-28ff-4fb3-833d-68bc06689385-kube-api-access-cb66t\") pod \"cert-manager-55b4d954d5-zsb6m\" (UID: \"a4cb5dce-28ff-4fb3-833d-68bc06689385\") " pod="cert-manager/cert-manager-55b4d954d5-zsb6m" Jul 06 21:03:44.228402 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:44.228309 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-zsb6m" Jul 06 21:03:44.370699 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:44.370668 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-zsb6m"] Jul 06 21:03:44.372577 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:03:44.372549 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4cb5dce_28ff_4fb3_833d_68bc06689385.slice/crio-6919d51f00382d786e30a2d6543a2a7f95a3bf5d101b85275130176aa049ccf8 WatchSource:0}: Error finding container 6919d51f00382d786e30a2d6543a2a7f95a3bf5d101b85275130176aa049ccf8: Status 404 returned error can't find the container with id 6919d51f00382d786e30a2d6543a2a7f95a3bf5d101b85275130176aa049ccf8 Jul 06 21:03:44.510082 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:44.509937 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-zsb6m" event={"ID":"a4cb5dce-28ff-4fb3-833d-68bc06689385","Type":"ContainerStarted","Data":"b10fb93c241ff1b2aff8bb00fb0b239e6efa8916dacd711fb9b5e6593021a734"} Jul 06 21:03:44.510082 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:44.509992 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-zsb6m" event={"ID":"a4cb5dce-28ff-4fb3-833d-68bc06689385","Type":"ContainerStarted","Data":"6919d51f00382d786e30a2d6543a2a7f95a3bf5d101b85275130176aa049ccf8"} Jul 06 21:03:44.531100 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:44.531044 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-zsb6m" podStartSLOduration=1.531025445 podStartE2EDuration="1.531025445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:03:44.529110275 +0000 UTC m=+308.847333535" watchObservedRunningTime="2026-07-06 21:03:44.531025445 +0000 UTC m=+308.849248696" Jul 06 21:03:46.897643 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:46.897600 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2lwrf"] Jul 06 21:03:46.900198 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:46.900165 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2lwrf" Jul 06 21:03:46.903574 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:46.903552 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 21:03:46.903689 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:46.903552 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 21:03:46.903689 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:46.903680 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-hkw6k\"" Jul 06 21:03:46.915204 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:46.915176 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2lwrf"] Jul 06 21:03:47.037908 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:47.037878 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/631e469d-d1e5-475b-9b77-6f5d16b5a61d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-2lwrf\" (UID: \"631e469d-d1e5-475b-9b77-6f5d16b5a61d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2lwrf" Jul 06 21:03:47.038080 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:47.037985 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6xt8\" (UniqueName: \"kubernetes.io/projected/631e469d-d1e5-475b-9b77-6f5d16b5a61d-kube-api-access-m6xt8\") pod \"openshift-lws-operator-fd8ccff4c-2lwrf\" (UID: \"631e469d-d1e5-475b-9b77-6f5d16b5a61d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2lwrf" Jul 06 21:03:47.138661 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:47.138620 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6xt8\" (UniqueName: \"kubernetes.io/projected/631e469d-d1e5-475b-9b77-6f5d16b5a61d-kube-api-access-m6xt8\") pod \"openshift-lws-operator-fd8ccff4c-2lwrf\" (UID: \"631e469d-d1e5-475b-9b77-6f5d16b5a61d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2lwrf" Jul 06 21:03:47.138856 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:47.138690 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/631e469d-d1e5-475b-9b77-6f5d16b5a61d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-2lwrf\" (UID: \"631e469d-d1e5-475b-9b77-6f5d16b5a61d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2lwrf" Jul 06 21:03:47.139229 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:47.139206 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/631e469d-d1e5-475b-9b77-6f5d16b5a61d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-2lwrf\" (UID: \"631e469d-d1e5-475b-9b77-6f5d16b5a61d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2lwrf" Jul 06 21:03:47.151314 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:47.151256 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6xt8\" (UniqueName: \"kubernetes.io/projected/631e469d-d1e5-475b-9b77-6f5d16b5a61d-kube-api-access-m6xt8\") pod \"openshift-lws-operator-fd8ccff4c-2lwrf\" (UID: \"631e469d-d1e5-475b-9b77-6f5d16b5a61d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2lwrf" Jul 06 21:03:47.209140 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:47.209105 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2lwrf" Jul 06 21:03:47.367408 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:47.367369 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2lwrf"] Jul 06 21:03:47.369757 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:03:47.369701 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod631e469d_d1e5_475b_9b77_6f5d16b5a61d.slice/crio-2a1721cb06f3a504c5ce94a8978a9be923d6f6f2a2cb6344f9a4ebb8bcc297ab WatchSource:0}: Error finding container 2a1721cb06f3a504c5ce94a8978a9be923d6f6f2a2cb6344f9a4ebb8bcc297ab: Status 404 returned error can't find the container with id 2a1721cb06f3a504c5ce94a8978a9be923d6f6f2a2cb6344f9a4ebb8bcc297ab Jul 06 21:03:47.371623 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:47.371600 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 21:03:47.522033 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:47.521916 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2lwrf" event={"ID":"631e469d-d1e5-475b-9b77-6f5d16b5a61d","Type":"ContainerStarted","Data":"2a1721cb06f3a504c5ce94a8978a9be923d6f6f2a2cb6344f9a4ebb8bcc297ab"} Jul 06 21:03:50.537195 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:50.537159 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2lwrf" event={"ID":"631e469d-d1e5-475b-9b77-6f5d16b5a61d","Type":"ContainerStarted","Data":"cfcf28f451bc23eb35c84cfd2156e1d83b03ac6a798da0622a3153c062c97c1f"} Jul 06 21:03:50.558455 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:03:50.558398 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2lwrf" podStartSLOduration=2.098365701 podStartE2EDuration="4.558376646s" totalImagesPullingTime="2.460010945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:03:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:03:50.557501258 +0000 UTC m=+314.875724520" watchObservedRunningTime="2026-07-06 21:03:50.558376646 +0000 UTC m=+314.876599906" Jul 06 21:04:07.042390 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:07.042357 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bd9f97986-7ht5r"] Jul 06 21:04:07.048763 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:07.048736 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5bd9f97986-7ht5r" Jul 06 21:04:07.052114 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:07.052087 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 21:04:07.052509 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:07.052488 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 21:04:07.052595 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:07.052577 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 21:04:07.052800 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:07.052783 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2tlc7\"" Jul 06 21:04:07.052910 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:07.052889 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 21:04:07.072518 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:07.072484 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bd9f97986-7ht5r"] Jul 06 21:04:07.223399 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:07.223354 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d-tmp\") pod \"opendatahub-operator-controller-manager-5bd9f97986-7ht5r\" (UID: \"bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d\") " pod="opendatahub/opendatahub-operator-controller-manager-5bd9f97986-7ht5r" Jul 06 21:04:07.223399 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:07.223404 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bd9f97986-7ht5r\" (UID: \"bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d\") " pod="opendatahub/opendatahub-operator-controller-manager-5bd9f97986-7ht5r" Jul 06 21:04:07.223641 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:07.223508 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d-manifests\") pod \"opendatahub-operator-controller-manager-5bd9f97986-7ht5r\" (UID: \"bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d\") " pod="opendatahub/opendatahub-operator-controller-manager-5bd9f97986-7ht5r" Jul 06 21:04:07.223641 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:07.223584 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bd9f97986-7ht5r\" (UID: \"bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d\") " pod="opendatahub/opendatahub-operator-controller-manager-5bd9f97986-7ht5r" Jul 06 21:04:07.223641 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:07.223634 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk8ls\" (UniqueName: \"kubernetes.io/projected/bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d-kube-api-access-tk8ls\") pod \"opendatahub-operator-controller-manager-5bd9f97986-7ht5r\" (UID: \"bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d\") " pod="opendatahub/opendatahub-operator-controller-manager-5bd9f97986-7ht5r" Jul 06 21:04:07.324797 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:07.324714 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d-tmp\") pod \"opendatahub-operator-controller-manager-5bd9f97986-7ht5r\" (UID: \"bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d\") " pod="opendatahub/opendatahub-operator-controller-manager-5bd9f97986-7ht5r" Jul 06 21:04:07.324797 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:07.324752 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bd9f97986-7ht5r\" (UID: \"bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d\") " pod="opendatahub/opendatahub-operator-controller-manager-5bd9f97986-7ht5r" Jul 06 21:04:07.324797 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:07.324793 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d-manifests\") pod \"opendatahub-operator-controller-manager-5bd9f97986-7ht5r\" (UID: \"bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d\") " pod="opendatahub/opendatahub-operator-controller-manager-5bd9f97986-7ht5r" Jul 06 21:04:07.325104 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:07.324836 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bd9f97986-7ht5r\" (UID: \"bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d\") " pod="opendatahub/opendatahub-operator-controller-manager-5bd9f97986-7ht5r" Jul 06 21:04:07.325104 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:07.324855 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk8ls\" (UniqueName: \"kubernetes.io/projected/bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d-kube-api-access-tk8ls\") pod \"opendatahub-operator-controller-manager-5bd9f97986-7ht5r\" (UID: \"bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d\") " pod="opendatahub/opendatahub-operator-controller-manager-5bd9f97986-7ht5r" Jul 06 21:04:07.325286 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:07.325259 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d-manifests\") pod \"opendatahub-operator-controller-manager-5bd9f97986-7ht5r\" (UID: \"bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d\") " pod="opendatahub/opendatahub-operator-controller-manager-5bd9f97986-7ht5r" Jul 06 21:04:07.327276 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:07.327247 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d-tmp\") pod \"opendatahub-operator-controller-manager-5bd9f97986-7ht5r\" (UID: \"bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d\") " pod="opendatahub/opendatahub-operator-controller-manager-5bd9f97986-7ht5r" Jul 06 21:04:07.327473 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:07.327451 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bd9f97986-7ht5r\" (UID: \"bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d\") " pod="opendatahub/opendatahub-operator-controller-manager-5bd9f97986-7ht5r" Jul 06 21:04:07.327517 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:07.327506 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bd9f97986-7ht5r\" (UID: \"bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d\") " pod="opendatahub/opendatahub-operator-controller-manager-5bd9f97986-7ht5r" Jul 06 21:04:07.338737 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:07.338705 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk8ls\" (UniqueName: \"kubernetes.io/projected/bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d-kube-api-access-tk8ls\") pod \"opendatahub-operator-controller-manager-5bd9f97986-7ht5r\" (UID: \"bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d\") " pod="opendatahub/opendatahub-operator-controller-manager-5bd9f97986-7ht5r" Jul 06 21:04:07.360466 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:07.360430 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5bd9f97986-7ht5r" Jul 06 21:04:07.518961 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:07.518934 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bd9f97986-7ht5r"] Jul 06 21:04:07.521022 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:04:07.520979 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd7ecdf_36af_47c7_8ca2_df1be45c7c1d.slice/crio-5707c75f88c0a6d3a1fb1bad73faf593826ca06a315933699b499533face97df WatchSource:0}: Error finding container 5707c75f88c0a6d3a1fb1bad73faf593826ca06a315933699b499533face97df: Status 404 returned error can't find the container with id 5707c75f88c0a6d3a1fb1bad73faf593826ca06a315933699b499533face97df Jul 06 21:04:07.607975 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:07.607879 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bd9f97986-7ht5r" event={"ID":"bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d","Type":"ContainerStarted","Data":"5707c75f88c0a6d3a1fb1bad73faf593826ca06a315933699b499533face97df"} Jul 06 21:04:11.631463 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:11.631420 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bd9f97986-7ht5r" event={"ID":"bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d","Type":"ContainerStarted","Data":"09fe808bc775aef5769c9e044f3cc75c2574ce57f554b07424e237ffe49da10c"} Jul 06 21:04:12.637638 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:12.637600 2577 generic.go:356] "Generic (PLEG): container finished" podID="bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d" containerID="09fe808bc775aef5769c9e044f3cc75c2574ce57f554b07424e237ffe49da10c" exitCode=0 Jul 06 21:04:12.638074 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:12.637697 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bd9f97986-7ht5r" event={"ID":"bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d","Type":"ContainerDied","Data":"09fe808bc775aef5769c9e044f3cc75c2574ce57f554b07424e237ffe49da10c"} Jul 06 21:04:13.644796 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:13.644763 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bd9f97986-7ht5r" event={"ID":"bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d","Type":"ContainerStarted","Data":"32b555224f9e1105e07aa5fd06feaa86fbac9d54342a42aff388c8de7520cdcc"} Jul 06 21:04:13.645177 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:13.644906 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5bd9f97986-7ht5r" Jul 06 21:04:13.671221 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:13.671162 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5bd9f97986-7ht5r" podStartSLOduration=1.5313655480000001 podStartE2EDuration="6.671146049s" totalImagesPullingTime="4.010968581s" totalInitContainerRuntime="1.12881192s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:04:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:04:13.669124485 +0000 UTC m=+337.987347762" watchObservedRunningTime="2026-07-06 21:04:13.671146049 +0000 UTC m=+337.989369308" Jul 06 21:04:24.652229 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:24.652192 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5bd9f97986-7ht5r" Jul 06 21:04:26.256928 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:26.256892 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cfbcfb449-tv2zz"] Jul 06 21:04:26.265974 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:26.265940 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6cfbcfb449-tv2zz" Jul 06 21:04:26.268439 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:26.268412 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-vln87\"" Jul 06 21:04:26.269420 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:26.269397 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 06 21:04:26.269557 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:26.269467 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 06 21:04:26.269557 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:26.269495 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 06 21:04:26.281320 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:26.281292 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cfbcfb449-tv2zz"] Jul 06 21:04:26.392365 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:26.392326 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/49ec5a9c-ddd1-4a95-9674-b02878be9d25-manager-config\") pod \"lws-controller-manager-6cfbcfb449-tv2zz\" (UID: \"49ec5a9c-ddd1-4a95-9674-b02878be9d25\") " pod="openshift-lws-operator/lws-controller-manager-6cfbcfb449-tv2zz" Jul 06 21:04:26.392365 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:26.392367 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/49ec5a9c-ddd1-4a95-9674-b02878be9d25-metrics-cert\") pod \"lws-controller-manager-6cfbcfb449-tv2zz\" (UID: \"49ec5a9c-ddd1-4a95-9674-b02878be9d25\") " pod="openshift-lws-operator/lws-controller-manager-6cfbcfb449-tv2zz" Jul 06 21:04:26.392595 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:26.392394 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62zsw\" (UniqueName: \"kubernetes.io/projected/49ec5a9c-ddd1-4a95-9674-b02878be9d25-kube-api-access-62zsw\") pod \"lws-controller-manager-6cfbcfb449-tv2zz\" (UID: \"49ec5a9c-ddd1-4a95-9674-b02878be9d25\") " pod="openshift-lws-operator/lws-controller-manager-6cfbcfb449-tv2zz" Jul 06 21:04:26.392595 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:26.392450 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49ec5a9c-ddd1-4a95-9674-b02878be9d25-cert\") pod \"lws-controller-manager-6cfbcfb449-tv2zz\" (UID: \"49ec5a9c-ddd1-4a95-9674-b02878be9d25\") " pod="openshift-lws-operator/lws-controller-manager-6cfbcfb449-tv2zz" Jul 06 21:04:26.493577 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:26.493537 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62zsw\" (UniqueName: \"kubernetes.io/projected/49ec5a9c-ddd1-4a95-9674-b02878be9d25-kube-api-access-62zsw\") pod \"lws-controller-manager-6cfbcfb449-tv2zz\" (UID: \"49ec5a9c-ddd1-4a95-9674-b02878be9d25\") " pod="openshift-lws-operator/lws-controller-manager-6cfbcfb449-tv2zz" Jul 06 21:04:26.493577 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:26.493578 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49ec5a9c-ddd1-4a95-9674-b02878be9d25-cert\") pod \"lws-controller-manager-6cfbcfb449-tv2zz\" (UID: \"49ec5a9c-ddd1-4a95-9674-b02878be9d25\") " pod="openshift-lws-operator/lws-controller-manager-6cfbcfb449-tv2zz" Jul 06 21:04:26.493799 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:26.493662 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/49ec5a9c-ddd1-4a95-9674-b02878be9d25-manager-config\") pod \"lws-controller-manager-6cfbcfb449-tv2zz\" (UID: \"49ec5a9c-ddd1-4a95-9674-b02878be9d25\") " pod="openshift-lws-operator/lws-controller-manager-6cfbcfb449-tv2zz" Jul 06 21:04:26.493799 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:26.493679 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/49ec5a9c-ddd1-4a95-9674-b02878be9d25-metrics-cert\") pod \"lws-controller-manager-6cfbcfb449-tv2zz\" (UID: \"49ec5a9c-ddd1-4a95-9674-b02878be9d25\") " pod="openshift-lws-operator/lws-controller-manager-6cfbcfb449-tv2zz" Jul 06 21:04:26.494510 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:26.494478 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/49ec5a9c-ddd1-4a95-9674-b02878be9d25-manager-config\") pod \"lws-controller-manager-6cfbcfb449-tv2zz\" (UID: \"49ec5a9c-ddd1-4a95-9674-b02878be9d25\") " pod="openshift-lws-operator/lws-controller-manager-6cfbcfb449-tv2zz" Jul 06 21:04:26.496247 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:26.496220 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49ec5a9c-ddd1-4a95-9674-b02878be9d25-cert\") pod \"lws-controller-manager-6cfbcfb449-tv2zz\" (UID: \"49ec5a9c-ddd1-4a95-9674-b02878be9d25\") " pod="openshift-lws-operator/lws-controller-manager-6cfbcfb449-tv2zz" Jul 06 21:04:26.496373 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:26.496270 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/49ec5a9c-ddd1-4a95-9674-b02878be9d25-metrics-cert\") pod \"lws-controller-manager-6cfbcfb449-tv2zz\" (UID: \"49ec5a9c-ddd1-4a95-9674-b02878be9d25\") " pod="openshift-lws-operator/lws-controller-manager-6cfbcfb449-tv2zz" Jul 06 21:04:26.507089 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:26.506981 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62zsw\" (UniqueName: \"kubernetes.io/projected/49ec5a9c-ddd1-4a95-9674-b02878be9d25-kube-api-access-62zsw\") pod \"lws-controller-manager-6cfbcfb449-tv2zz\" (UID: \"49ec5a9c-ddd1-4a95-9674-b02878be9d25\") " pod="openshift-lws-operator/lws-controller-manager-6cfbcfb449-tv2zz" Jul 06 21:04:26.576203 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:26.576163 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6cfbcfb449-tv2zz" Jul 06 21:04:26.807072 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:04:26.807034 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ec5a9c_ddd1_4a95_9674_b02878be9d25.slice/crio-df0165d501f873bc01a36f4ebc8bb4e6d35d1171484c546677d0c9b4e24076fc WatchSource:0}: Error finding container df0165d501f873bc01a36f4ebc8bb4e6d35d1171484c546677d0c9b4e24076fc: Status 404 returned error can't find the container with id df0165d501f873bc01a36f4ebc8bb4e6d35d1171484c546677d0c9b4e24076fc Jul 06 21:04:26.878898 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:26.878864 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cfbcfb449-tv2zz"] Jul 06 21:04:27.709348 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:27.709273 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6cfbcfb449-tv2zz" event={"ID":"49ec5a9c-ddd1-4a95-9674-b02878be9d25","Type":"ContainerStarted","Data":"df0165d501f873bc01a36f4ebc8bb4e6d35d1171484c546677d0c9b4e24076fc"} Jul 06 21:04:28.715912 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:28.715878 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6cfbcfb449-tv2zz" event={"ID":"49ec5a9c-ddd1-4a95-9674-b02878be9d25","Type":"ContainerStarted","Data":"075dca3c9ad41f7ee9934b0b5edba3803989c734e016991d8ad832071c8bab12"} Jul 06 21:04:28.716351 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:28.715986 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6cfbcfb449-tv2zz" Jul 06 21:04:39.722858 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:39.722820 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6cfbcfb449-tv2zz" Jul 06 21:04:39.761292 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:39.761225 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6cfbcfb449-tv2zz" podStartSLOduration=12.098404642 podStartE2EDuration="13.761200999s" totalImagesPullingTime="1.662796357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:04:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:04:28.738647893 +0000 UTC m=+353.056871153" watchObservedRunningTime="2026-07-06 21:04:39.761200999 +0000 UTC m=+364.079424259" Jul 06 21:04:45.861616 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:45.861572 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb"] Jul 06 21:04:45.870268 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:45.870234 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:45.872915 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:45.872884 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 21:04:45.873109 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:45.872931 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-rk79w\"" Jul 06 21:04:45.873109 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:45.872936 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 21:04:45.873109 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:45.872982 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 21:04:45.881827 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:45.881798 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb"] Jul 06 21:04:45.973799 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:45.973757 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/980321c5-434c-4e39-ae74-2c4c5b1537b8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jr2lb\" (UID: \"980321c5-434c-4e39-ae74-2c4c5b1537b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:45.974055 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:45.973807 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/980321c5-434c-4e39-ae74-2c4c5b1537b8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jr2lb\" (UID: \"980321c5-434c-4e39-ae74-2c4c5b1537b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:45.974055 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:45.973896 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/980321c5-434c-4e39-ae74-2c4c5b1537b8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jr2lb\" (UID: \"980321c5-434c-4e39-ae74-2c4c5b1537b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:45.974055 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:45.973938 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/980321c5-434c-4e39-ae74-2c4c5b1537b8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jr2lb\" (UID: \"980321c5-434c-4e39-ae74-2c4c5b1537b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:45.974243 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:45.974065 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/980321c5-434c-4e39-ae74-2c4c5b1537b8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jr2lb\" (UID: \"980321c5-434c-4e39-ae74-2c4c5b1537b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:45.974243 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:45.974101 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/980321c5-434c-4e39-ae74-2c4c5b1537b8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jr2lb\" (UID: \"980321c5-434c-4e39-ae74-2c4c5b1537b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:45.974243 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:45.974129 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/980321c5-434c-4e39-ae74-2c4c5b1537b8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jr2lb\" (UID: \"980321c5-434c-4e39-ae74-2c4c5b1537b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:45.974243 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:45.974191 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmhpb\" (UniqueName: \"kubernetes.io/projected/980321c5-434c-4e39-ae74-2c4c5b1537b8-kube-api-access-bmhpb\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jr2lb\" (UID: \"980321c5-434c-4e39-ae74-2c4c5b1537b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:45.974398 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:45.974249 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/980321c5-434c-4e39-ae74-2c4c5b1537b8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jr2lb\" (UID: \"980321c5-434c-4e39-ae74-2c4c5b1537b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:46.075296 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:46.075254 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/980321c5-434c-4e39-ae74-2c4c5b1537b8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jr2lb\" (UID: \"980321c5-434c-4e39-ae74-2c4c5b1537b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:46.075522 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:46.075312 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmhpb\" (UniqueName: \"kubernetes.io/projected/980321c5-434c-4e39-ae74-2c4c5b1537b8-kube-api-access-bmhpb\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jr2lb\" (UID: \"980321c5-434c-4e39-ae74-2c4c5b1537b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:46.075522 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:46.075346 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/980321c5-434c-4e39-ae74-2c4c5b1537b8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jr2lb\" (UID: \"980321c5-434c-4e39-ae74-2c4c5b1537b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:46.075522 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:46.075382 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/980321c5-434c-4e39-ae74-2c4c5b1537b8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jr2lb\" (UID: \"980321c5-434c-4e39-ae74-2c4c5b1537b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:46.075681 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:46.075534 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/980321c5-434c-4e39-ae74-2c4c5b1537b8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jr2lb\" (UID: \"980321c5-434c-4e39-ae74-2c4c5b1537b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:46.075681 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:46.075602 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/980321c5-434c-4e39-ae74-2c4c5b1537b8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jr2lb\" (UID: \"980321c5-434c-4e39-ae74-2c4c5b1537b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:46.075681 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:46.075651 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/980321c5-434c-4e39-ae74-2c4c5b1537b8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jr2lb\" (UID: \"980321c5-434c-4e39-ae74-2c4c5b1537b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:46.075827 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:46.075716 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/980321c5-434c-4e39-ae74-2c4c5b1537b8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jr2lb\" (UID: \"980321c5-434c-4e39-ae74-2c4c5b1537b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:46.075827 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:46.075747 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/980321c5-434c-4e39-ae74-2c4c5b1537b8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jr2lb\" (UID: \"980321c5-434c-4e39-ae74-2c4c5b1537b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:46.075927 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:46.075824 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/980321c5-434c-4e39-ae74-2c4c5b1537b8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jr2lb\" (UID: \"980321c5-434c-4e39-ae74-2c4c5b1537b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:46.076036 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:46.075991 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/980321c5-434c-4e39-ae74-2c4c5b1537b8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jr2lb\" (UID: \"980321c5-434c-4e39-ae74-2c4c5b1537b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:46.076176 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:46.076157 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/980321c5-434c-4e39-ae74-2c4c5b1537b8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jr2lb\" (UID: \"980321c5-434c-4e39-ae74-2c4c5b1537b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:46.076265 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:46.076210 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/980321c5-434c-4e39-ae74-2c4c5b1537b8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jr2lb\" (UID: \"980321c5-434c-4e39-ae74-2c4c5b1537b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:46.076265 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:46.076251 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/980321c5-434c-4e39-ae74-2c4c5b1537b8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jr2lb\" (UID: \"980321c5-434c-4e39-ae74-2c4c5b1537b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:46.078082 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:46.078051 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/980321c5-434c-4e39-ae74-2c4c5b1537b8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jr2lb\" (UID: \"980321c5-434c-4e39-ae74-2c4c5b1537b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:46.078329 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:46.078309 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/980321c5-434c-4e39-ae74-2c4c5b1537b8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jr2lb\" (UID: \"980321c5-434c-4e39-ae74-2c4c5b1537b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:46.089520 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:46.089489 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/980321c5-434c-4e39-ae74-2c4c5b1537b8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jr2lb\" (UID: \"980321c5-434c-4e39-ae74-2c4c5b1537b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:46.089670 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:46.089551 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmhpb\" (UniqueName: \"kubernetes.io/projected/980321c5-434c-4e39-ae74-2c4c5b1537b8-kube-api-access-bmhpb\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4jr2lb\" (UID: \"980321c5-434c-4e39-ae74-2c4c5b1537b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:46.182703 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:46.182655 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:46.350442 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:46.350380 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb"] Jul 06 21:04:46.352696 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:04:46.352670 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod980321c5_434c_4e39_ae74_2c4c5b1537b8.slice/crio-2aca607df2119a1094e914eb836d99e0ae69f3e7c2e015d99c5e55377f4ba8c1 WatchSource:0}: Error finding container 2aca607df2119a1094e914eb836d99e0ae69f3e7c2e015d99c5e55377f4ba8c1: Status 404 returned error can't find the container with id 2aca607df2119a1094e914eb836d99e0ae69f3e7c2e015d99c5e55377f4ba8c1 Jul 06 21:04:46.799360 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:46.799324 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" event={"ID":"980321c5-434c-4e39-ae74-2c4c5b1537b8","Type":"ContainerStarted","Data":"2aca607df2119a1094e914eb836d99e0ae69f3e7c2e015d99c5e55377f4ba8c1"} Jul 06 21:04:49.438141 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:49.438093 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:04:49.438473 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:49.438172 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:04:49.438473 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:49.438202 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:04:49.817093 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:49.816985 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" event={"ID":"980321c5-434c-4e39-ae74-2c4c5b1537b8","Type":"ContainerStarted","Data":"5f738f594ad5fbf4266a7e53184b2dc204094d3bd4f92140e081b81a71103fba"} Jul 06 21:04:49.842704 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:49.842622 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" podStartSLOduration=1.759601059 podStartE2EDuration="4.842602366s" totalImagesPullingTime="3.083001307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:04:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:04:49.840809854 +0000 UTC m=+374.159033115" watchObservedRunningTime="2026-07-06 21:04:49.842602366 +0000 UTC m=+374.160825625" Jul 06 21:04:50.183797 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:50.183755 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:50.188531 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:50.188499 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:50.822822 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:50.822791 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:04:50.823860 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:04:50.823841 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4jr2lb" Jul 06 21:05:28.059099 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:28.059064 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fk5s8"] Jul 06 21:05:28.069504 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:28.069477 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fk5s8" Jul 06 21:05:28.074577 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:28.074549 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 21:05:28.074775 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:28.074549 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 21:05:28.075169 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:28.075140 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fk5s8"] Jul 06 21:05:28.075422 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:28.075403 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-4btxd\"" Jul 06 21:05:28.155414 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:28.155377 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mvjj\" (UniqueName: \"kubernetes.io/projected/ca99f467-c443-4b0a-bb38-7c0660f99753-kube-api-access-5mvjj\") pod \"kuadrant-operator-catalog-fk5s8\" (UID: \"ca99f467-c443-4b0a-bb38-7c0660f99753\") " pod="kuadrant-system/kuadrant-operator-catalog-fk5s8" Jul 06 21:05:28.256226 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:28.256189 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mvjj\" (UniqueName: \"kubernetes.io/projected/ca99f467-c443-4b0a-bb38-7c0660f99753-kube-api-access-5mvjj\") pod \"kuadrant-operator-catalog-fk5s8\" (UID: \"ca99f467-c443-4b0a-bb38-7c0660f99753\") " pod="kuadrant-system/kuadrant-operator-catalog-fk5s8" Jul 06 21:05:28.269927 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:28.269890 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mvjj\" (UniqueName: \"kubernetes.io/projected/ca99f467-c443-4b0a-bb38-7c0660f99753-kube-api-access-5mvjj\") pod \"kuadrant-operator-catalog-fk5s8\" (UID: \"ca99f467-c443-4b0a-bb38-7c0660f99753\") " pod="kuadrant-system/kuadrant-operator-catalog-fk5s8" Jul 06 21:05:28.381878 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:28.381847 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fk5s8" Jul 06 21:05:28.394145 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:28.394116 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fk5s8"] Jul 06 21:05:28.527195 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:28.527153 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fk5s8"] Jul 06 21:05:28.530195 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:05:28.530161 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca99f467_c443_4b0a_bb38_7c0660f99753.slice/crio-3fa89638574e697c8e72b46e1fc7d77eb35b9e35e85fcec2ace1ebc93fcf4b71 WatchSource:0}: Error finding container 3fa89638574e697c8e72b46e1fc7d77eb35b9e35e85fcec2ace1ebc93fcf4b71: Status 404 returned error can't find the container with id 3fa89638574e697c8e72b46e1fc7d77eb35b9e35e85fcec2ace1ebc93fcf4b71 Jul 06 21:05:28.603718 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:28.603680 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rtw2l"] Jul 06 21:05:28.608556 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:28.608530 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rtw2l" Jul 06 21:05:28.617650 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:28.617623 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rtw2l"] Jul 06 21:05:28.762260 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:28.762166 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k449w\" (UniqueName: \"kubernetes.io/projected/b5119452-a047-4283-9b62-b1f37fe2032c-kube-api-access-k449w\") pod \"kuadrant-operator-catalog-rtw2l\" (UID: \"b5119452-a047-4283-9b62-b1f37fe2032c\") " pod="kuadrant-system/kuadrant-operator-catalog-rtw2l" Jul 06 21:05:28.862679 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:28.862643 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k449w\" (UniqueName: \"kubernetes.io/projected/b5119452-a047-4283-9b62-b1f37fe2032c-kube-api-access-k449w\") pod \"kuadrant-operator-catalog-rtw2l\" (UID: \"b5119452-a047-4283-9b62-b1f37fe2032c\") " pod="kuadrant-system/kuadrant-operator-catalog-rtw2l" Jul 06 21:05:28.873484 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:28.873446 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k449w\" (UniqueName: \"kubernetes.io/projected/b5119452-a047-4283-9b62-b1f37fe2032c-kube-api-access-k449w\") pod \"kuadrant-operator-catalog-rtw2l\" (UID: \"b5119452-a047-4283-9b62-b1f37fe2032c\") " pod="kuadrant-system/kuadrant-operator-catalog-rtw2l" Jul 06 21:05:28.918532 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:28.918491 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rtw2l" Jul 06 21:05:28.988224 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:28.988187 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fk5s8" event={"ID":"ca99f467-c443-4b0a-bb38-7c0660f99753","Type":"ContainerStarted","Data":"3fa89638574e697c8e72b46e1fc7d77eb35b9e35e85fcec2ace1ebc93fcf4b71"} Jul 06 21:05:29.075961 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:29.075933 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rtw2l"] Jul 06 21:05:29.104781 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:05:29.104743 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5119452_a047_4283_9b62_b1f37fe2032c.slice/crio-fea76bc4d3e2f0face60f1be4a52a0ada7ff5f829f37137becd750e730a8c0a8 WatchSource:0}: Error finding container fea76bc4d3e2f0face60f1be4a52a0ada7ff5f829f37137becd750e730a8c0a8: Status 404 returned error can't find the container with id fea76bc4d3e2f0face60f1be4a52a0ada7ff5f829f37137becd750e730a8c0a8 Jul 06 21:05:29.996320 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:29.996280 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rtw2l" event={"ID":"b5119452-a047-4283-9b62-b1f37fe2032c","Type":"ContainerStarted","Data":"fea76bc4d3e2f0face60f1be4a52a0ada7ff5f829f37137becd750e730a8c0a8"} Jul 06 21:05:31.002794 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:31.002748 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rtw2l" event={"ID":"b5119452-a047-4283-9b62-b1f37fe2032c","Type":"ContainerStarted","Data":"39c99bb01e49a6492e9fd64eaf5aab3dd974e175d1c0ecea64aa51fd332dd674"} Jul 06 21:05:31.004420 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:31.004381 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fk5s8" event={"ID":"ca99f467-c443-4b0a-bb38-7c0660f99753","Type":"ContainerStarted","Data":"52b9f96a18e654ddcc407a348b615caeb0f54025540fa8e5d71e354f1e62027b"} Jul 06 21:05:31.004563 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:31.004441 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-fk5s8" podUID="ca99f467-c443-4b0a-bb38-7c0660f99753" containerName="registry-server" containerID="cri-o://52b9f96a18e654ddcc407a348b615caeb0f54025540fa8e5d71e354f1e62027b" gracePeriod=2 Jul 06 21:05:31.034085 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:31.033964 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-rtw2l" podStartSLOduration=1.352458388 podStartE2EDuration="3.033946606s" totalImagesPullingTime="1.681488218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:05:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:05:31.033239923 +0000 UTC m=+415.351463198" watchObservedRunningTime="2026-07-06 21:05:31.033946606 +0000 UTC m=+415.352169895" Jul 06 21:05:31.065357 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:31.065299 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-fk5s8" podStartSLOduration=0.812193472 podStartE2EDuration="3.065282127s" totalImagesPullingTime="2.253088655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:05:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:05:31.064929971 +0000 UTC m=+415.383153253" watchObservedRunningTime="2026-07-06 21:05:31.065282127 +0000 UTC m=+415.383505387" Jul 06 21:05:31.245884 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:31.245854 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fk5s8" Jul 06 21:05:31.389649 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:31.389599 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mvjj\" (UniqueName: \"kubernetes.io/projected/ca99f467-c443-4b0a-bb38-7c0660f99753-kube-api-access-5mvjj\") pod \"ca99f467-c443-4b0a-bb38-7c0660f99753\" (UID: \"ca99f467-c443-4b0a-bb38-7c0660f99753\") " Jul 06 21:05:31.391954 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:31.391916 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca99f467-c443-4b0a-bb38-7c0660f99753-kube-api-access-5mvjj" (OuterVolumeSpecName: "kube-api-access-5mvjj") pod "ca99f467-c443-4b0a-bb38-7c0660f99753" (UID: "ca99f467-c443-4b0a-bb38-7c0660f99753"). InnerVolumeSpecName "kube-api-access-5mvjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:05:31.490405 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:31.490371 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mvjj\" (UniqueName: \"kubernetes.io/projected/ca99f467-c443-4b0a-bb38-7c0660f99753-kube-api-access-5mvjj\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:05:32.010106 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:32.010068 2577 generic.go:356] "Generic (PLEG): container finished" podID="ca99f467-c443-4b0a-bb38-7c0660f99753" containerID="52b9f96a18e654ddcc407a348b615caeb0f54025540fa8e5d71e354f1e62027b" exitCode=0 Jul 06 21:05:32.010509 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:32.010126 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fk5s8" Jul 06 21:05:32.010509 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:32.010152 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fk5s8" event={"ID":"ca99f467-c443-4b0a-bb38-7c0660f99753","Type":"ContainerDied","Data":"52b9f96a18e654ddcc407a348b615caeb0f54025540fa8e5d71e354f1e62027b"} Jul 06 21:05:32.010509 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:32.010230 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fk5s8" event={"ID":"ca99f467-c443-4b0a-bb38-7c0660f99753","Type":"ContainerDied","Data":"3fa89638574e697c8e72b46e1fc7d77eb35b9e35e85fcec2ace1ebc93fcf4b71"} Jul 06 21:05:32.010509 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:32.010251 2577 scope.go:117] "RemoveContainer" containerID="52b9f96a18e654ddcc407a348b615caeb0f54025540fa8e5d71e354f1e62027b" Jul 06 21:05:32.019453 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:32.019433 2577 scope.go:117] "RemoveContainer" containerID="52b9f96a18e654ddcc407a348b615caeb0f54025540fa8e5d71e354f1e62027b" Jul 06 21:05:32.019788 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:05:32.019769 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52b9f96a18e654ddcc407a348b615caeb0f54025540fa8e5d71e354f1e62027b\": container with ID starting with 52b9f96a18e654ddcc407a348b615caeb0f54025540fa8e5d71e354f1e62027b not found: ID does not exist" containerID="52b9f96a18e654ddcc407a348b615caeb0f54025540fa8e5d71e354f1e62027b" Jul 06 21:05:32.019839 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:32.019798 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52b9f96a18e654ddcc407a348b615caeb0f54025540fa8e5d71e354f1e62027b"} err="failed to get container status \"52b9f96a18e654ddcc407a348b615caeb0f54025540fa8e5d71e354f1e62027b\": rpc error: code = NotFound desc = could not find container \"52b9f96a18e654ddcc407a348b615caeb0f54025540fa8e5d71e354f1e62027b\": container with ID starting with 52b9f96a18e654ddcc407a348b615caeb0f54025540fa8e5d71e354f1e62027b not found: ID does not exist" Jul 06 21:05:32.035261 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:32.035229 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fk5s8"] Jul 06 21:05:32.040319 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:32.040291 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fk5s8"] Jul 06 21:05:32.168383 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:32.168336 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca99f467-c443-4b0a-bb38-7c0660f99753" path="/var/lib/kubelet/pods/ca99f467-c443-4b0a-bb38-7c0660f99753/volumes" Jul 06 21:05:38.918677 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:38.918626 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-rtw2l" Jul 06 21:05:38.918677 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:38.918692 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-rtw2l" Jul 06 21:05:38.942078 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:38.942048 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-rtw2l" Jul 06 21:05:39.067874 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:39.067841 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-rtw2l" Jul 06 21:05:59.265057 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:59.264966 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-wtnzk"] Jul 06 21:05:59.265435 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:59.265342 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca99f467-c443-4b0a-bb38-7c0660f99753" containerName="registry-server" Jul 06 21:05:59.265435 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:59.265361 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca99f467-c443-4b0a-bb38-7c0660f99753" containerName="registry-server" Jul 06 21:05:59.265503 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:59.265443 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca99f467-c443-4b0a-bb38-7c0660f99753" containerName="registry-server" Jul 06 21:05:59.268345 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:59.268322 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-wtnzk" Jul 06 21:05:59.271573 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:59.271550 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-gpgpp\"" Jul 06 21:05:59.287681 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:59.287643 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-wtnzk"] Jul 06 21:05:59.343332 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:59.343285 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw6kd\" (UniqueName: \"kubernetes.io/projected/04a627be-8e89-407d-9065-5ede85ce9e8a-kube-api-access-dw6kd\") pod \"authorino-operator-75cc8d5cb7-wtnzk\" (UID: \"04a627be-8e89-407d-9065-5ede85ce9e8a\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-wtnzk" Jul 06 21:05:59.444529 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:59.444480 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw6kd\" (UniqueName: \"kubernetes.io/projected/04a627be-8e89-407d-9065-5ede85ce9e8a-kube-api-access-dw6kd\") pod \"authorino-operator-75cc8d5cb7-wtnzk\" (UID: \"04a627be-8e89-407d-9065-5ede85ce9e8a\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-wtnzk" Jul 06 21:05:59.456348 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:59.456300 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw6kd\" (UniqueName: \"kubernetes.io/projected/04a627be-8e89-407d-9065-5ede85ce9e8a-kube-api-access-dw6kd\") pod \"authorino-operator-75cc8d5cb7-wtnzk\" (UID: \"04a627be-8e89-407d-9065-5ede85ce9e8a\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-wtnzk" Jul 06 21:05:59.580251 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:59.580160 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-wtnzk" Jul 06 21:05:59.745187 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:05:59.744954 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-wtnzk"] Jul 06 21:05:59.748204 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:05:59.748167 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04a627be_8e89_407d_9065_5ede85ce9e8a.slice/crio-c5851a4f692f767b5ca7a3cdcdb8c4407d5c8c98620d22f1ed9124acdd1ca9ee WatchSource:0}: Error finding container c5851a4f692f767b5ca7a3cdcdb8c4407d5c8c98620d22f1ed9124acdd1ca9ee: Status 404 returned error can't find the container with id c5851a4f692f767b5ca7a3cdcdb8c4407d5c8c98620d22f1ed9124acdd1ca9ee Jul 06 21:06:00.146208 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:00.146170 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-wtnzk" event={"ID":"04a627be-8e89-407d-9065-5ede85ce9e8a","Type":"ContainerStarted","Data":"c5851a4f692f767b5ca7a3cdcdb8c4407d5c8c98620d22f1ed9124acdd1ca9ee"} Jul 06 21:06:03.161823 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:03.161785 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-wtnzk" event={"ID":"04a627be-8e89-407d-9065-5ede85ce9e8a","Type":"ContainerStarted","Data":"602204a451c7fd6e8ef6eb67ebeb9b272843aacffb9af0407db6286426597616"} Jul 06 21:06:03.162356 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:03.161910 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-wtnzk" Jul 06 21:06:03.185697 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:03.185639 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-wtnzk" podStartSLOduration=1.4639045689999999 podStartE2EDuration="4.185620473s" totalImagesPullingTime="2.721715904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:05:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:06:03.184342951 +0000 UTC m=+447.502566224" watchObservedRunningTime="2026-07-06 21:06:03.185620473 +0000 UTC m=+447.503843733" Jul 06 21:06:07.224672 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:07.224636 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qblxg"] Jul 06 21:06:07.231337 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:07.231312 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qblxg" Jul 06 21:06:07.234239 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:07.234212 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kvxqb\"" Jul 06 21:06:07.243851 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:07.243819 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qblxg"] Jul 06 21:06:07.320196 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:07.320165 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5v4x\" (UniqueName: \"kubernetes.io/projected/eaff8f27-a091-4166-ac3e-20478b0c1134-kube-api-access-p5v4x\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qblxg\" (UID: \"eaff8f27-a091-4166-ac3e-20478b0c1134\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qblxg" Jul 06 21:06:07.320358 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:07.320201 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eaff8f27-a091-4166-ac3e-20478b0c1134-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qblxg\" (UID: \"eaff8f27-a091-4166-ac3e-20478b0c1134\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qblxg" Jul 06 21:06:07.421403 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:07.421360 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eaff8f27-a091-4166-ac3e-20478b0c1134-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qblxg\" (UID: \"eaff8f27-a091-4166-ac3e-20478b0c1134\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qblxg" Jul 06 21:06:07.421569 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:07.421468 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5v4x\" (UniqueName: \"kubernetes.io/projected/eaff8f27-a091-4166-ac3e-20478b0c1134-kube-api-access-p5v4x\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qblxg\" (UID: \"eaff8f27-a091-4166-ac3e-20478b0c1134\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qblxg" Jul 06 21:06:07.421770 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:07.421748 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eaff8f27-a091-4166-ac3e-20478b0c1134-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qblxg\" (UID: \"eaff8f27-a091-4166-ac3e-20478b0c1134\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qblxg" Jul 06 21:06:07.431841 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:07.431813 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5v4x\" (UniqueName: \"kubernetes.io/projected/eaff8f27-a091-4166-ac3e-20478b0c1134-kube-api-access-p5v4x\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qblxg\" (UID: \"eaff8f27-a091-4166-ac3e-20478b0c1134\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qblxg" Jul 06 21:06:07.542542 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:07.542441 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qblxg" Jul 06 21:06:07.696262 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:07.696237 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qblxg"] Jul 06 21:06:08.185968 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:08.185930 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qblxg" event={"ID":"eaff8f27-a091-4166-ac3e-20478b0c1134","Type":"ContainerStarted","Data":"9b912f6dc162243db335868f6323b5c50f607f32f211c51764eb46acd390a320"} Jul 06 21:06:14.169262 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:14.169234 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-wtnzk" Jul 06 21:06:16.229417 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:16.229377 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qblxg" event={"ID":"eaff8f27-a091-4166-ac3e-20478b0c1134","Type":"ContainerStarted","Data":"0be20ba99dff916ecdaecc04e3d425612556d44711d8998784b1cb57ad3b2509"} Jul 06 21:06:16.229846 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:16.229514 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qblxg" Jul 06 21:06:16.251458 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:16.251340 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qblxg" podStartSLOduration=1.322950888 podStartE2EDuration="9.251320542s" totalImagesPullingTime="7.928369654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:06:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:06:16.249567193 +0000 UTC m=+460.567790452" watchObservedRunningTime="2026-07-06 21:06:16.251320542 +0000 UTC m=+460.569543803" Jul 06 21:06:27.237818 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:27.237787 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qblxg" Jul 06 21:06:29.191717 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:29.191676 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qblxg"] Jul 06 21:06:29.192168 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:29.191869 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qblxg" podUID="eaff8f27-a091-4166-ac3e-20478b0c1134" containerName="manager" containerID="cri-o://0be20ba99dff916ecdaecc04e3d425612556d44711d8998784b1cb57ad3b2509" gracePeriod=2 Jul 06 21:06:29.205873 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:29.205836 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qblxg"] Jul 06 21:06:29.208332 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:29.208298 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wtcmf"] Jul 06 21:06:29.208981 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:29.208956 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaff8f27-a091-4166-ac3e-20478b0c1134" containerName="manager" Jul 06 21:06:29.208981 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:29.208983 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaff8f27-a091-4166-ac3e-20478b0c1134" containerName="manager" Jul 06 21:06:29.209137 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:29.209102 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaff8f27-a091-4166-ac3e-20478b0c1134" containerName="manager" Jul 06 21:06:29.212613 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:29.212589 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wtcmf" Jul 06 21:06:29.219888 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:06:29.219843 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qblxg\" is forbidden: User \"system:node:ip-10-0-136-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-207.ec2.internal' and this object" podUID="eaff8f27-a091-4166-ac3e-20478b0c1134" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qblxg" Jul 06 21:06:29.235766 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:29.235729 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wtcmf"] Jul 06 21:06:29.315812 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:29.315776 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b45f1869-1981-40c2-be6f-833398aaf1e2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-wtcmf\" (UID: \"b45f1869-1981-40c2-be6f-833398aaf1e2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wtcmf" Jul 06 21:06:29.315963 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:29.315866 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mfkr\" (UniqueName: \"kubernetes.io/projected/b45f1869-1981-40c2-be6f-833398aaf1e2-kube-api-access-7mfkr\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-wtcmf\" (UID: \"b45f1869-1981-40c2-be6f-833398aaf1e2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wtcmf" Jul 06 21:06:29.417283 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:29.417248 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b45f1869-1981-40c2-be6f-833398aaf1e2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-wtcmf\" (UID: \"b45f1869-1981-40c2-be6f-833398aaf1e2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wtcmf" Jul 06 21:06:29.417469 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:29.417309 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mfkr\" (UniqueName: \"kubernetes.io/projected/b45f1869-1981-40c2-be6f-833398aaf1e2-kube-api-access-7mfkr\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-wtcmf\" (UID: \"b45f1869-1981-40c2-be6f-833398aaf1e2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wtcmf" Jul 06 21:06:29.417807 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:29.417694 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b45f1869-1981-40c2-be6f-833398aaf1e2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-wtcmf\" (UID: \"b45f1869-1981-40c2-be6f-833398aaf1e2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wtcmf" Jul 06 21:06:29.430597 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:29.430569 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mfkr\" (UniqueName: \"kubernetes.io/projected/b45f1869-1981-40c2-be6f-833398aaf1e2-kube-api-access-7mfkr\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-wtcmf\" (UID: \"b45f1869-1981-40c2-be6f-833398aaf1e2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wtcmf" Jul 06 21:06:29.449562 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:29.449492 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qblxg" Jul 06 21:06:29.452054 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:06:29.451993 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qblxg\" is forbidden: User \"system:node:ip-10-0-136-207.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-207.ec2.internal' and this object" podUID="eaff8f27-a091-4166-ac3e-20478b0c1134" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qblxg" Jul 06 21:06:29.517792 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:29.517752 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5v4x\" (UniqueName: \"kubernetes.io/projected/eaff8f27-a091-4166-ac3e-20478b0c1134-kube-api-access-p5v4x\") pod \"eaff8f27-a091-4166-ac3e-20478b0c1134\" (UID: \"eaff8f27-a091-4166-ac3e-20478b0c1134\") " Jul 06 21:06:29.517957 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:29.517827 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eaff8f27-a091-4166-ac3e-20478b0c1134-extensions-socket-volume\") pod \"eaff8f27-a091-4166-ac3e-20478b0c1134\" (UID: \"eaff8f27-a091-4166-ac3e-20478b0c1134\") " Jul 06 21:06:29.518403 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:29.518371 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaff8f27-a091-4166-ac3e-20478b0c1134-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "eaff8f27-a091-4166-ac3e-20478b0c1134" (UID: "eaff8f27-a091-4166-ac3e-20478b0c1134"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:06:29.519931 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:29.519908 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaff8f27-a091-4166-ac3e-20478b0c1134-kube-api-access-p5v4x" (OuterVolumeSpecName: "kube-api-access-p5v4x") pod "eaff8f27-a091-4166-ac3e-20478b0c1134" (UID: "eaff8f27-a091-4166-ac3e-20478b0c1134"). InnerVolumeSpecName "kube-api-access-p5v4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:06:29.575538 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:29.575497 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wtcmf" Jul 06 21:06:29.619037 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:29.618987 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eaff8f27-a091-4166-ac3e-20478b0c1134-extensions-socket-volume\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:06:29.619037 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:29.619035 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5v4x\" (UniqueName: \"kubernetes.io/projected/eaff8f27-a091-4166-ac3e-20478b0c1134-kube-api-access-p5v4x\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:06:29.724395 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:29.724367 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wtcmf"] Jul 06 21:06:29.727024 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:06:29.726974 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb45f1869_1981_40c2_be6f_833398aaf1e2.slice/crio-0385f97814fd6d2f8ebc9187401b3f1f8e59150a3f99acce0ddd3dae71bc4ff5 WatchSource:0}: Error finding container 0385f97814fd6d2f8ebc9187401b3f1f8e59150a3f99acce0ddd3dae71bc4ff5: Status 404 returned error can't find the container with id 0385f97814fd6d2f8ebc9187401b3f1f8e59150a3f99acce0ddd3dae71bc4ff5 Jul 06 21:06:30.174665 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:30.174627 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eaff8f27-a091-4166-ac3e-20478b0c1134" path="/var/lib/kubelet/pods/eaff8f27-a091-4166-ac3e-20478b0c1134/volumes" Jul 06 21:06:30.298277 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:30.298236 2577 generic.go:356] "Generic (PLEG): container finished" podID="eaff8f27-a091-4166-ac3e-20478b0c1134" containerID="0be20ba99dff916ecdaecc04e3d425612556d44711d8998784b1cb57ad3b2509" exitCode=0 Jul 06 21:06:30.298735 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:30.298298 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qblxg" Jul 06 21:06:30.298735 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:30.298346 2577 scope.go:117] "RemoveContainer" containerID="0be20ba99dff916ecdaecc04e3d425612556d44711d8998784b1cb57ad3b2509" Jul 06 21:06:30.300349 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:30.300304 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wtcmf" event={"ID":"b45f1869-1981-40c2-be6f-833398aaf1e2","Type":"ContainerStarted","Data":"6ab6fc5911c49fd832a98f50eab16648fee3b3ae15ca9427afc377e76a7a094e"} Jul 06 21:06:30.300454 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:30.300374 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wtcmf" event={"ID":"b45f1869-1981-40c2-be6f-833398aaf1e2","Type":"ContainerStarted","Data":"0385f97814fd6d2f8ebc9187401b3f1f8e59150a3f99acce0ddd3dae71bc4ff5"} Jul 06 21:06:30.300681 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:30.300637 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wtcmf" Jul 06 21:06:30.307677 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:30.307654 2577 scope.go:117] "RemoveContainer" containerID="0be20ba99dff916ecdaecc04e3d425612556d44711d8998784b1cb57ad3b2509" Jul 06 21:06:30.307996 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:06:30.307971 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0be20ba99dff916ecdaecc04e3d425612556d44711d8998784b1cb57ad3b2509\": container with ID starting with 0be20ba99dff916ecdaecc04e3d425612556d44711d8998784b1cb57ad3b2509 not found: ID does not exist" containerID="0be20ba99dff916ecdaecc04e3d425612556d44711d8998784b1cb57ad3b2509" Jul 06 21:06:30.308087 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:30.308025 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0be20ba99dff916ecdaecc04e3d425612556d44711d8998784b1cb57ad3b2509"} err="failed to get container status \"0be20ba99dff916ecdaecc04e3d425612556d44711d8998784b1cb57ad3b2509\": rpc error: code = NotFound desc = could not find container \"0be20ba99dff916ecdaecc04e3d425612556d44711d8998784b1cb57ad3b2509\": container with ID starting with 0be20ba99dff916ecdaecc04e3d425612556d44711d8998784b1cb57ad3b2509 not found: ID does not exist" Jul 06 21:06:30.327710 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:30.327638 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wtcmf" podStartSLOduration=1.327615351 podStartE2EDuration="1.327615351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:06:30.323704708 +0000 UTC m=+474.641927979" watchObservedRunningTime="2026-07-06 21:06:30.327615351 +0000 UTC m=+474.645838612" Jul 06 21:06:41.308102 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:41.308066 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wtcmf" Jul 06 21:06:57.861957 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:57.861919 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd"] Jul 06 21:06:57.867174 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:57.867151 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:57.874307 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:57.874270 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-ws9wb\"" Jul 06 21:06:57.895834 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:57.895799 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd"] Jul 06 21:06:57.975287 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:57.975243 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90be9695-5aa7-4767-95a3-81acb10d1237-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-77zzd\" (UID: \"90be9695-5aa7-4767-95a3-81acb10d1237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:57.975287 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:57.975286 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90be9695-5aa7-4767-95a3-81acb10d1237-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-77zzd\" (UID: \"90be9695-5aa7-4767-95a3-81acb10d1237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:57.975515 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:57.975310 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90be9695-5aa7-4767-95a3-81acb10d1237-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-77zzd\" (UID: \"90be9695-5aa7-4767-95a3-81acb10d1237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:57.975515 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:57.975329 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90be9695-5aa7-4767-95a3-81acb10d1237-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-77zzd\" (UID: \"90be9695-5aa7-4767-95a3-81acb10d1237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:57.975515 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:57.975385 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90be9695-5aa7-4767-95a3-81acb10d1237-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-77zzd\" (UID: \"90be9695-5aa7-4767-95a3-81acb10d1237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:57.975515 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:57.975420 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90be9695-5aa7-4767-95a3-81acb10d1237-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-77zzd\" (UID: \"90be9695-5aa7-4767-95a3-81acb10d1237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:57.975515 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:57.975502 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90be9695-5aa7-4767-95a3-81acb10d1237-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-77zzd\" (UID: \"90be9695-5aa7-4767-95a3-81acb10d1237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:57.975670 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:57.975528 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90be9695-5aa7-4767-95a3-81acb10d1237-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-77zzd\" (UID: \"90be9695-5aa7-4767-95a3-81acb10d1237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:57.975670 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:57.975559 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6wtn\" (UniqueName: \"kubernetes.io/projected/90be9695-5aa7-4767-95a3-81acb10d1237-kube-api-access-z6wtn\") pod \"maas-default-gateway-openshift-default-56654dcf7-77zzd\" (UID: \"90be9695-5aa7-4767-95a3-81acb10d1237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:58.076628 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:58.076589 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90be9695-5aa7-4767-95a3-81acb10d1237-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-77zzd\" (UID: \"90be9695-5aa7-4767-95a3-81acb10d1237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:58.076628 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:58.076633 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90be9695-5aa7-4767-95a3-81acb10d1237-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-77zzd\" (UID: \"90be9695-5aa7-4767-95a3-81acb10d1237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:58.076886 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:58.076666 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6wtn\" (UniqueName: \"kubernetes.io/projected/90be9695-5aa7-4767-95a3-81acb10d1237-kube-api-access-z6wtn\") pod \"maas-default-gateway-openshift-default-56654dcf7-77zzd\" (UID: \"90be9695-5aa7-4767-95a3-81acb10d1237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:58.076886 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:58.076694 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90be9695-5aa7-4767-95a3-81acb10d1237-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-77zzd\" (UID: \"90be9695-5aa7-4767-95a3-81acb10d1237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:58.076886 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:58.076710 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90be9695-5aa7-4767-95a3-81acb10d1237-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-77zzd\" (UID: \"90be9695-5aa7-4767-95a3-81acb10d1237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:58.076886 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:58.076730 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90be9695-5aa7-4767-95a3-81acb10d1237-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-77zzd\" (UID: \"90be9695-5aa7-4767-95a3-81acb10d1237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:58.076886 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:58.076756 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90be9695-5aa7-4767-95a3-81acb10d1237-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-77zzd\" (UID: \"90be9695-5aa7-4767-95a3-81acb10d1237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:58.076886 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:58.076795 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90be9695-5aa7-4767-95a3-81acb10d1237-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-77zzd\" (UID: \"90be9695-5aa7-4767-95a3-81acb10d1237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:58.076886 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:58.076823 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90be9695-5aa7-4767-95a3-81acb10d1237-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-77zzd\" (UID: \"90be9695-5aa7-4767-95a3-81acb10d1237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:58.077299 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:58.077184 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90be9695-5aa7-4767-95a3-81acb10d1237-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-77zzd\" (UID: \"90be9695-5aa7-4767-95a3-81acb10d1237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:58.077299 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:58.077213 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90be9695-5aa7-4767-95a3-81acb10d1237-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-77zzd\" (UID: \"90be9695-5aa7-4767-95a3-81acb10d1237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:58.077299 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:58.077261 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90be9695-5aa7-4767-95a3-81acb10d1237-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-77zzd\" (UID: \"90be9695-5aa7-4767-95a3-81acb10d1237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:58.077561 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:58.077541 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90be9695-5aa7-4767-95a3-81acb10d1237-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-77zzd\" (UID: \"90be9695-5aa7-4767-95a3-81acb10d1237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:58.077628 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:58.077559 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90be9695-5aa7-4767-95a3-81acb10d1237-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-77zzd\" (UID: \"90be9695-5aa7-4767-95a3-81acb10d1237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:58.079376 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:58.079355 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90be9695-5aa7-4767-95a3-81acb10d1237-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-77zzd\" (UID: \"90be9695-5aa7-4767-95a3-81acb10d1237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:58.079467 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:58.079442 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90be9695-5aa7-4767-95a3-81acb10d1237-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-77zzd\" (UID: \"90be9695-5aa7-4767-95a3-81acb10d1237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:58.086385 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:58.086352 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90be9695-5aa7-4767-95a3-81acb10d1237-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-77zzd\" (UID: \"90be9695-5aa7-4767-95a3-81acb10d1237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:58.086546 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:58.086471 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6wtn\" (UniqueName: \"kubernetes.io/projected/90be9695-5aa7-4767-95a3-81acb10d1237-kube-api-access-z6wtn\") pod \"maas-default-gateway-openshift-default-56654dcf7-77zzd\" (UID: \"90be9695-5aa7-4767-95a3-81acb10d1237\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:58.177590 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:58.177555 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:58.325944 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:58.325916 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd"] Jul 06 21:06:58.328238 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:06:58.328210 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90be9695_5aa7_4767_95a3_81acb10d1237.slice/crio-83582a7faab300bee56425d0525a69680312d32a198cef2ab971851ad7c60ee1 WatchSource:0}: Error finding container 83582a7faab300bee56425d0525a69680312d32a198cef2ab971851ad7c60ee1: Status 404 returned error can't find the container with id 83582a7faab300bee56425d0525a69680312d32a198cef2ab971851ad7c60ee1 Jul 06 21:06:58.330362 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:58.330334 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:06:58.330424 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:58.330402 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:06:58.330465 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:58.330433 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:06:58.436197 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:58.436071 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" event={"ID":"90be9695-5aa7-4767-95a3-81acb10d1237","Type":"ContainerStarted","Data":"c29cd1a039bc0cb6923a0c91846eaca929815e5e85a3ac03e4f21c1ed8480fdd"} Jul 06 21:06:58.436197 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:58.436167 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" event={"ID":"90be9695-5aa7-4767-95a3-81acb10d1237","Type":"ContainerStarted","Data":"83582a7faab300bee56425d0525a69680312d32a198cef2ab971851ad7c60ee1"} Jul 06 21:06:58.460732 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:58.460669 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" podStartSLOduration=1.460642798 podStartE2EDuration="1.460642798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:06:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:06:58.458863528 +0000 UTC m=+502.777086807" watchObservedRunningTime="2026-07-06 21:06:58.460642798 +0000 UTC m=+502.778866059" Jul 06 21:06:59.178356 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:59.178317 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:59.183237 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:59.183205 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:59.214600 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:59.214561 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wtcmf"] Jul 06 21:06:59.214798 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:59.214760 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wtcmf" podUID="b45f1869-1981-40c2-be6f-833398aaf1e2" containerName="manager" containerID="cri-o://6ab6fc5911c49fd832a98f50eab16648fee3b3ae15ca9427afc377e76a7a094e" gracePeriod=10 Jul 06 21:06:59.442457 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:59.442356 2577 generic.go:356] "Generic (PLEG): container finished" podID="b45f1869-1981-40c2-be6f-833398aaf1e2" containerID="6ab6fc5911c49fd832a98f50eab16648fee3b3ae15ca9427afc377e76a7a094e" exitCode=0 Jul 06 21:06:59.442457 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:59.442436 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wtcmf" event={"ID":"b45f1869-1981-40c2-be6f-833398aaf1e2","Type":"ContainerDied","Data":"6ab6fc5911c49fd832a98f50eab16648fee3b3ae15ca9427afc377e76a7a094e"} Jul 06 21:06:59.442804 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:59.442785 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:59.443824 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:59.443803 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:06:59.478525 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:59.478498 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wtcmf" Jul 06 21:06:59.590350 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:59.590313 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mfkr\" (UniqueName: \"kubernetes.io/projected/b45f1869-1981-40c2-be6f-833398aaf1e2-kube-api-access-7mfkr\") pod \"b45f1869-1981-40c2-be6f-833398aaf1e2\" (UID: \"b45f1869-1981-40c2-be6f-833398aaf1e2\") " Jul 06 21:06:59.590530 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:59.590417 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b45f1869-1981-40c2-be6f-833398aaf1e2-extensions-socket-volume\") pod \"b45f1869-1981-40c2-be6f-833398aaf1e2\" (UID: \"b45f1869-1981-40c2-be6f-833398aaf1e2\") " Jul 06 21:06:59.590823 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:59.590792 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45f1869-1981-40c2-be6f-833398aaf1e2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b45f1869-1981-40c2-be6f-833398aaf1e2" (UID: "b45f1869-1981-40c2-be6f-833398aaf1e2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:06:59.592555 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:59.592531 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45f1869-1981-40c2-be6f-833398aaf1e2-kube-api-access-7mfkr" (OuterVolumeSpecName: "kube-api-access-7mfkr") pod "b45f1869-1981-40c2-be6f-833398aaf1e2" (UID: "b45f1869-1981-40c2-be6f-833398aaf1e2"). InnerVolumeSpecName "kube-api-access-7mfkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:06:59.692073 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:59.691997 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mfkr\" (UniqueName: \"kubernetes.io/projected/b45f1869-1981-40c2-be6f-833398aaf1e2-kube-api-access-7mfkr\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:06:59.692073 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:06:59.692068 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b45f1869-1981-40c2-be6f-833398aaf1e2-extensions-socket-volume\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:07:00.448535 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:00.448442 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wtcmf" Jul 06 21:07:00.448535 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:00.448467 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wtcmf" event={"ID":"b45f1869-1981-40c2-be6f-833398aaf1e2","Type":"ContainerDied","Data":"0385f97814fd6d2f8ebc9187401b3f1f8e59150a3f99acce0ddd3dae71bc4ff5"} Jul 06 21:07:00.449050 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:00.448545 2577 scope.go:117] "RemoveContainer" containerID="6ab6fc5911c49fd832a98f50eab16648fee3b3ae15ca9427afc377e76a7a094e" Jul 06 21:07:00.477630 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:00.477594 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wtcmf"] Jul 06 21:07:00.486207 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:00.486176 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wtcmf"] Jul 06 21:07:02.168023 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:02.167967 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b45f1869-1981-40c2-be6f-833398aaf1e2" path="/var/lib/kubelet/pods/b45f1869-1981-40c2-be6f-833398aaf1e2/volumes" Jul 06 21:07:16.386454 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:16.386415 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-84ph6"] Jul 06 21:07:16.386939 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:16.386756 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b45f1869-1981-40c2-be6f-833398aaf1e2" containerName="manager" Jul 06 21:07:16.386939 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:16.386767 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b45f1869-1981-40c2-be6f-833398aaf1e2" containerName="manager" Jul 06 21:07:16.386939 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:16.386836 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b45f1869-1981-40c2-be6f-833398aaf1e2" containerName="manager" Jul 06 21:07:16.391370 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:16.391349 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-84ph6" Jul 06 21:07:16.394092 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:16.394064 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-f6h6j\"" Jul 06 21:07:16.394217 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:16.394069 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 06 21:07:16.402586 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:16.402556 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-84ph6"] Jul 06 21:07:16.436496 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:16.436460 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grdkh\" (UniqueName: \"kubernetes.io/projected/ca4db750-47ff-4e85-a60c-0c7a6dabd729-kube-api-access-grdkh\") pod \"limitador-limitador-64c94fd64f-84ph6\" (UID: \"ca4db750-47ff-4e85-a60c-0c7a6dabd729\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-84ph6" Jul 06 21:07:16.436681 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:16.436563 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ca4db750-47ff-4e85-a60c-0c7a6dabd729-config-file\") pod \"limitador-limitador-64c94fd64f-84ph6\" (UID: \"ca4db750-47ff-4e85-a60c-0c7a6dabd729\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-84ph6" Jul 06 21:07:16.481112 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:16.481076 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-84ph6"] Jul 06 21:07:16.537648 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:16.537607 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ca4db750-47ff-4e85-a60c-0c7a6dabd729-config-file\") pod \"limitador-limitador-64c94fd64f-84ph6\" (UID: \"ca4db750-47ff-4e85-a60c-0c7a6dabd729\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-84ph6" Jul 06 21:07:16.537854 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:16.537676 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grdkh\" (UniqueName: \"kubernetes.io/projected/ca4db750-47ff-4e85-a60c-0c7a6dabd729-kube-api-access-grdkh\") pod \"limitador-limitador-64c94fd64f-84ph6\" (UID: \"ca4db750-47ff-4e85-a60c-0c7a6dabd729\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-84ph6" Jul 06 21:07:16.538299 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:16.538266 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ca4db750-47ff-4e85-a60c-0c7a6dabd729-config-file\") pod \"limitador-limitador-64c94fd64f-84ph6\" (UID: \"ca4db750-47ff-4e85-a60c-0c7a6dabd729\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-84ph6" Jul 06 21:07:16.547148 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:16.547115 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grdkh\" (UniqueName: \"kubernetes.io/projected/ca4db750-47ff-4e85-a60c-0c7a6dabd729-kube-api-access-grdkh\") pod \"limitador-limitador-64c94fd64f-84ph6\" (UID: \"ca4db750-47ff-4e85-a60c-0c7a6dabd729\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-84ph6" Jul 06 21:07:16.702141 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:16.702043 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-84ph6" Jul 06 21:07:16.852328 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:16.852301 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-84ph6"] Jul 06 21:07:16.854996 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:07:16.854966 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca4db750_47ff_4e85_a60c_0c7a6dabd729.slice/crio-3d5b95a80cd60829a2612a5fbbc926c321de1bd80947eaef3a8ca424abc538a8 WatchSource:0}: Error finding container 3d5b95a80cd60829a2612a5fbbc926c321de1bd80947eaef3a8ca424abc538a8: Status 404 returned error can't find the container with id 3d5b95a80cd60829a2612a5fbbc926c321de1bd80947eaef3a8ca424abc538a8 Jul 06 21:07:17.488220 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:17.488184 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-96rps"] Jul 06 21:07:17.492167 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:17.492140 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-96rps" Jul 06 21:07:17.494934 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:17.494907 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-gclwr\"" Jul 06 21:07:17.498057 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:17.498030 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-96rps"] Jul 06 21:07:17.531505 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:17.531463 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-84ph6" event={"ID":"ca4db750-47ff-4e85-a60c-0c7a6dabd729","Type":"ContainerStarted","Data":"3d5b95a80cd60829a2612a5fbbc926c321de1bd80947eaef3a8ca424abc538a8"} Jul 06 21:07:17.547296 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:17.547254 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wfmr\" (UniqueName: \"kubernetes.io/projected/37938462-8eef-4675-b24c-9e6433fcf740-kube-api-access-7wfmr\") pod \"authorino-75d556bd77-96rps\" (UID: \"37938462-8eef-4675-b24c-9e6433fcf740\") " pod="kuadrant-system/authorino-75d556bd77-96rps" Jul 06 21:07:17.649731 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:17.649241 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wfmr\" (UniqueName: \"kubernetes.io/projected/37938462-8eef-4675-b24c-9e6433fcf740-kube-api-access-7wfmr\") pod \"authorino-75d556bd77-96rps\" (UID: \"37938462-8eef-4675-b24c-9e6433fcf740\") " pod="kuadrant-system/authorino-75d556bd77-96rps" Jul 06 21:07:17.661409 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:17.661373 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wfmr\" (UniqueName: \"kubernetes.io/projected/37938462-8eef-4675-b24c-9e6433fcf740-kube-api-access-7wfmr\") pod \"authorino-75d556bd77-96rps\" (UID: \"37938462-8eef-4675-b24c-9e6433fcf740\") " pod="kuadrant-system/authorino-75d556bd77-96rps" Jul 06 21:07:17.809042 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:17.808940 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-96rps" Jul 06 21:07:17.992095 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:17.991987 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-96rps"] Jul 06 21:07:17.995291 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:07:17.995251 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37938462_8eef_4675_b24c_9e6433fcf740.slice/crio-d06815c73e602fb84d9a90ef3ad8038aeb923c35d79dcbddd7faa288f99337b1 WatchSource:0}: Error finding container d06815c73e602fb84d9a90ef3ad8038aeb923c35d79dcbddd7faa288f99337b1: Status 404 returned error can't find the container with id d06815c73e602fb84d9a90ef3ad8038aeb923c35d79dcbddd7faa288f99337b1 Jul 06 21:07:18.540023 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:18.539942 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-96rps" event={"ID":"37938462-8eef-4675-b24c-9e6433fcf740","Type":"ContainerStarted","Data":"d06815c73e602fb84d9a90ef3ad8038aeb923c35d79dcbddd7faa288f99337b1"} Jul 06 21:07:22.564112 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:22.564076 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-96rps" event={"ID":"37938462-8eef-4675-b24c-9e6433fcf740","Type":"ContainerStarted","Data":"ef4ca5d8ba094d9875ae6425e99381aae40756e4b4a7d3cc5c002c40fc21b049"} Jul 06 21:07:22.565706 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:22.565676 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-84ph6" event={"ID":"ca4db750-47ff-4e85-a60c-0c7a6dabd729","Type":"ContainerStarted","Data":"d7796c2f14395e290daa2cd5d38eef1c4c79e22f8399587067ea5081ce57a545"} Jul 06 21:07:22.565836 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:22.565733 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-84ph6" Jul 06 21:07:22.587438 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:22.587380 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-96rps" podStartSLOduration=1.787337239 podStartE2EDuration="5.587360763s" totalImagesPullingTime="3.800023524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:07:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:07:22.586788861 +0000 UTC m=+526.905012145" watchObservedRunningTime="2026-07-06 21:07:22.587360763 +0000 UTC m=+526.905584022" Jul 06 21:07:22.624731 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:22.620342 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-84ph6" podStartSLOduration=1.6873725419999999 podStartE2EDuration="6.620312336s" totalImagesPullingTime="4.932939794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:07:22.618258516 +0000 UTC m=+526.936481774" watchObservedRunningTime="2026-07-06 21:07:22.620312336 +0000 UTC m=+526.938535597" Jul 06 21:07:33.464194 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:33.464105 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-84ph6"] Jul 06 21:07:33.464606 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:33.464312 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-84ph6" podUID="ca4db750-47ff-4e85-a60c-0c7a6dabd729" containerName="limitador" containerID="cri-o://d7796c2f14395e290daa2cd5d38eef1c4c79e22f8399587067ea5081ce57a545" gracePeriod=30 Jul 06 21:07:33.465090 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:33.465041 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-84ph6" Jul 06 21:07:34.026559 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:34.026487 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-84ph6" Jul 06 21:07:34.106381 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:34.106339 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ca4db750-47ff-4e85-a60c-0c7a6dabd729-config-file\") pod \"ca4db750-47ff-4e85-a60c-0c7a6dabd729\" (UID: \"ca4db750-47ff-4e85-a60c-0c7a6dabd729\") " Jul 06 21:07:34.106533 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:34.106422 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grdkh\" (UniqueName: \"kubernetes.io/projected/ca4db750-47ff-4e85-a60c-0c7a6dabd729-kube-api-access-grdkh\") pod \"ca4db750-47ff-4e85-a60c-0c7a6dabd729\" (UID: \"ca4db750-47ff-4e85-a60c-0c7a6dabd729\") " Jul 06 21:07:34.106716 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:34.106690 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca4db750-47ff-4e85-a60c-0c7a6dabd729-config-file" (OuterVolumeSpecName: "config-file") pod "ca4db750-47ff-4e85-a60c-0c7a6dabd729" (UID: "ca4db750-47ff-4e85-a60c-0c7a6dabd729"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:07:34.108464 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:34.108439 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca4db750-47ff-4e85-a60c-0c7a6dabd729-kube-api-access-grdkh" (OuterVolumeSpecName: "kube-api-access-grdkh") pod "ca4db750-47ff-4e85-a60c-0c7a6dabd729" (UID: "ca4db750-47ff-4e85-a60c-0c7a6dabd729"). InnerVolumeSpecName "kube-api-access-grdkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:07:34.207616 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:34.207581 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grdkh\" (UniqueName: \"kubernetes.io/projected/ca4db750-47ff-4e85-a60c-0c7a6dabd729-kube-api-access-grdkh\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:07:34.207616 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:34.207612 2577 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ca4db750-47ff-4e85-a60c-0c7a6dabd729-config-file\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:07:34.629584 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:34.629548 2577 generic.go:356] "Generic (PLEG): container finished" podID="ca4db750-47ff-4e85-a60c-0c7a6dabd729" containerID="d7796c2f14395e290daa2cd5d38eef1c4c79e22f8399587067ea5081ce57a545" exitCode=0 Jul 06 21:07:34.630069 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:34.629592 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-84ph6" Jul 06 21:07:34.630069 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:34.629632 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-84ph6" event={"ID":"ca4db750-47ff-4e85-a60c-0c7a6dabd729","Type":"ContainerDied","Data":"d7796c2f14395e290daa2cd5d38eef1c4c79e22f8399587067ea5081ce57a545"} Jul 06 21:07:34.630069 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:34.629682 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-84ph6" event={"ID":"ca4db750-47ff-4e85-a60c-0c7a6dabd729","Type":"ContainerDied","Data":"3d5b95a80cd60829a2612a5fbbc926c321de1bd80947eaef3a8ca424abc538a8"} Jul 06 21:07:34.630069 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:34.629706 2577 scope.go:117] "RemoveContainer" containerID="d7796c2f14395e290daa2cd5d38eef1c4c79e22f8399587067ea5081ce57a545" Jul 06 21:07:34.638433 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:34.638412 2577 scope.go:117] "RemoveContainer" containerID="d7796c2f14395e290daa2cd5d38eef1c4c79e22f8399587067ea5081ce57a545" Jul 06 21:07:34.638717 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:07:34.638697 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7796c2f14395e290daa2cd5d38eef1c4c79e22f8399587067ea5081ce57a545\": container with ID starting with d7796c2f14395e290daa2cd5d38eef1c4c79e22f8399587067ea5081ce57a545 not found: ID does not exist" containerID="d7796c2f14395e290daa2cd5d38eef1c4c79e22f8399587067ea5081ce57a545" Jul 06 21:07:34.638806 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:34.638728 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7796c2f14395e290daa2cd5d38eef1c4c79e22f8399587067ea5081ce57a545"} err="failed to get container status \"d7796c2f14395e290daa2cd5d38eef1c4c79e22f8399587067ea5081ce57a545\": rpc error: code = NotFound desc = could not find container \"d7796c2f14395e290daa2cd5d38eef1c4c79e22f8399587067ea5081ce57a545\": container with ID starting with d7796c2f14395e290daa2cd5d38eef1c4c79e22f8399587067ea5081ce57a545 not found: ID does not exist" Jul 06 21:07:34.652772 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:34.652733 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-84ph6"] Jul 06 21:07:34.657869 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:34.657838 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-84ph6"] Jul 06 21:07:36.170787 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:36.170758 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca4db750-47ff-4e85-a60c-0c7a6dabd729" path="/var/lib/kubelet/pods/ca4db750-47ff-4e85-a60c-0c7a6dabd729/volumes" Jul 06 21:07:36.686454 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:36.686419 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-r9pjt"] Jul 06 21:07:36.686812 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:36.686799 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca4db750-47ff-4e85-a60c-0c7a6dabd729" containerName="limitador" Jul 06 21:07:36.686855 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:36.686814 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca4db750-47ff-4e85-a60c-0c7a6dabd729" containerName="limitador" Jul 06 21:07:36.686889 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:36.686878 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca4db750-47ff-4e85-a60c-0c7a6dabd729" containerName="limitador" Jul 06 21:07:36.691473 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:36.691446 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-r9pjt" Jul 06 21:07:36.694026 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:36.693990 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\"" Jul 06 21:07:36.694155 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:36.693997 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9bjl5\"" Jul 06 21:07:36.700961 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:36.700934 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-r9pjt"] Jul 06 21:07:36.831749 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:36.831712 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlkfh\" (UniqueName: \"kubernetes.io/projected/fdc5952f-0f86-4a51-8678-d1887ab25ac0-kube-api-access-jlkfh\") pod \"postgres-69fb4594cf-r9pjt\" (UID: \"fdc5952f-0f86-4a51-8678-d1887ab25ac0\") " pod="opendatahub/postgres-69fb4594cf-r9pjt" Jul 06 21:07:36.831939 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:36.831768 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fdc5952f-0f86-4a51-8678-d1887ab25ac0-data\") pod \"postgres-69fb4594cf-r9pjt\" (UID: \"fdc5952f-0f86-4a51-8678-d1887ab25ac0\") " pod="opendatahub/postgres-69fb4594cf-r9pjt" Jul 06 21:07:36.932906 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:36.932867 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlkfh\" (UniqueName: \"kubernetes.io/projected/fdc5952f-0f86-4a51-8678-d1887ab25ac0-kube-api-access-jlkfh\") pod \"postgres-69fb4594cf-r9pjt\" (UID: \"fdc5952f-0f86-4a51-8678-d1887ab25ac0\") " pod="opendatahub/postgres-69fb4594cf-r9pjt" Jul 06 21:07:36.933099 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:36.932930 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fdc5952f-0f86-4a51-8678-d1887ab25ac0-data\") pod \"postgres-69fb4594cf-r9pjt\" (UID: \"fdc5952f-0f86-4a51-8678-d1887ab25ac0\") " pod="opendatahub/postgres-69fb4594cf-r9pjt" Jul 06 21:07:36.933292 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:36.933276 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fdc5952f-0f86-4a51-8678-d1887ab25ac0-data\") pod \"postgres-69fb4594cf-r9pjt\" (UID: \"fdc5952f-0f86-4a51-8678-d1887ab25ac0\") " pod="opendatahub/postgres-69fb4594cf-r9pjt" Jul 06 21:07:36.944826 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:36.944743 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlkfh\" (UniqueName: \"kubernetes.io/projected/fdc5952f-0f86-4a51-8678-d1887ab25ac0-kube-api-access-jlkfh\") pod \"postgres-69fb4594cf-r9pjt\" (UID: \"fdc5952f-0f86-4a51-8678-d1887ab25ac0\") " pod="opendatahub/postgres-69fb4594cf-r9pjt" Jul 06 21:07:37.002433 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:37.002396 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-r9pjt" Jul 06 21:07:37.142137 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:37.142107 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-r9pjt"] Jul 06 21:07:37.143908 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:07:37.143883 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc5952f_0f86_4a51_8678_d1887ab25ac0.slice/crio-01892a629fdca4779e5d67ca8f2fa463dda7f0b94eecd64bb48c0d5e656e3ef8 WatchSource:0}: Error finding container 01892a629fdca4779e5d67ca8f2fa463dda7f0b94eecd64bb48c0d5e656e3ef8: Status 404 returned error can't find the container with id 01892a629fdca4779e5d67ca8f2fa463dda7f0b94eecd64bb48c0d5e656e3ef8 Jul 06 21:07:37.646035 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:37.645971 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-r9pjt" event={"ID":"fdc5952f-0f86-4a51-8678-d1887ab25ac0","Type":"ContainerStarted","Data":"01892a629fdca4779e5d67ca8f2fa463dda7f0b94eecd64bb48c0d5e656e3ef8"} Jul 06 21:07:43.677508 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:43.677469 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-r9pjt" event={"ID":"fdc5952f-0f86-4a51-8678-d1887ab25ac0","Type":"ContainerStarted","Data":"c169e4a89238f5138b7a735be4f9eddec57cb9349955466bbbb8b454a5e87318"} Jul 06 21:07:43.677909 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:43.677597 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-r9pjt" Jul 06 21:07:49.714273 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:49.714221 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-r9pjt" Jul 06 21:07:49.735637 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:49.735567 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-r9pjt" podStartSLOduration=8.02942487 podStartE2EDuration="13.735549124s" totalImagesPullingTime="5.706124254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:07:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:07:43.702327139 +0000 UTC m=+548.020550404" watchObservedRunningTime="2026-07-06 21:07:49.735549124 +0000 UTC m=+554.053772384" Jul 06 21:07:50.312294 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:50.312250 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bfg7b"] Jul 06 21:07:50.320316 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:50.320283 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-bfg7b" Jul 06 21:07:50.324960 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:50.324927 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bfg7b"] Jul 06 21:07:50.362716 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:50.362674 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb482\" (UniqueName: \"kubernetes.io/projected/c1601a80-baef-418f-9118-c5a7a0451d71-kube-api-access-jb482\") pod \"authorino-5d4b67c446-bfg7b\" (UID: \"c1601a80-baef-418f-9118-c5a7a0451d71\") " pod="kuadrant-system/authorino-5d4b67c446-bfg7b" Jul 06 21:07:50.463357 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:50.463316 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb482\" (UniqueName: \"kubernetes.io/projected/c1601a80-baef-418f-9118-c5a7a0451d71-kube-api-access-jb482\") pod \"authorino-5d4b67c446-bfg7b\" (UID: \"c1601a80-baef-418f-9118-c5a7a0451d71\") " pod="kuadrant-system/authorino-5d4b67c446-bfg7b" Jul 06 21:07:50.476183 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:50.476150 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb482\" (UniqueName: \"kubernetes.io/projected/c1601a80-baef-418f-9118-c5a7a0451d71-kube-api-access-jb482\") pod \"authorino-5d4b67c446-bfg7b\" (UID: \"c1601a80-baef-418f-9118-c5a7a0451d71\") " pod="kuadrant-system/authorino-5d4b67c446-bfg7b" Jul 06 21:07:50.556379 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:50.556338 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bfg7b"] Jul 06 21:07:50.556634 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:50.556621 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-bfg7b" Jul 06 21:07:50.590474 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:50.590425 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-66569b5d8f-8kzcm"] Jul 06 21:07:50.595995 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:50.595962 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66569b5d8f-8kzcm" Jul 06 21:07:50.601422 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:50.601392 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66569b5d8f-8kzcm"] Jul 06 21:07:50.667854 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:50.667816 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc6j5\" (UniqueName: \"kubernetes.io/projected/2d1e7bd3-8de7-4f61-a0b3-1619561f8b02-kube-api-access-kc6j5\") pod \"authorino-66569b5d8f-8kzcm\" (UID: \"2d1e7bd3-8de7-4f61-a0b3-1619561f8b02\") " pod="kuadrant-system/authorino-66569b5d8f-8kzcm" Jul 06 21:07:50.715213 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:50.715183 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bfg7b"] Jul 06 21:07:50.715607 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:07:50.715586 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1601a80_baef_418f_9118_c5a7a0451d71.slice/crio-d8f09e88a2f5097c2b7e8e1c6d09d6d1b6f3ee1885245b279b2befccdb42d802 WatchSource:0}: Error finding container d8f09e88a2f5097c2b7e8e1c6d09d6d1b6f3ee1885245b279b2befccdb42d802: Status 404 returned error can't find the container with id d8f09e88a2f5097c2b7e8e1c6d09d6d1b6f3ee1885245b279b2befccdb42d802 Jul 06 21:07:50.768647 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:50.768594 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc6j5\" (UniqueName: \"kubernetes.io/projected/2d1e7bd3-8de7-4f61-a0b3-1619561f8b02-kube-api-access-kc6j5\") pod \"authorino-66569b5d8f-8kzcm\" (UID: \"2d1e7bd3-8de7-4f61-a0b3-1619561f8b02\") " pod="kuadrant-system/authorino-66569b5d8f-8kzcm" Jul 06 21:07:50.788054 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:50.788000 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc6j5\" (UniqueName: \"kubernetes.io/projected/2d1e7bd3-8de7-4f61-a0b3-1619561f8b02-kube-api-access-kc6j5\") pod \"authorino-66569b5d8f-8kzcm\" (UID: \"2d1e7bd3-8de7-4f61-a0b3-1619561f8b02\") " pod="kuadrant-system/authorino-66569b5d8f-8kzcm" Jul 06 21:07:50.908859 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:50.908818 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66569b5d8f-8kzcm" Jul 06 21:07:51.048879 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:51.048851 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66569b5d8f-8kzcm"] Jul 06 21:07:51.050845 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:07:51.050819 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d1e7bd3_8de7_4f61_a0b3_1619561f8b02.slice/crio-bbe1cfe216478d8284bec526b8fe0c8b4ec5e1525b4114807982363831293a6d WatchSource:0}: Error finding container bbe1cfe216478d8284bec526b8fe0c8b4ec5e1525b4114807982363831293a6d: Status 404 returned error can't find the container with id bbe1cfe216478d8284bec526b8fe0c8b4ec5e1525b4114807982363831293a6d Jul 06 21:07:51.717788 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:51.717737 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66569b5d8f-8kzcm" event={"ID":"2d1e7bd3-8de7-4f61-a0b3-1619561f8b02","Type":"ContainerStarted","Data":"195b5708b0404ece2bf77035dbfb9e35fcabc132ffd6e2bdc8a2adb07ff95951"} Jul 06 21:07:51.717788 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:51.717794 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66569b5d8f-8kzcm" event={"ID":"2d1e7bd3-8de7-4f61-a0b3-1619561f8b02","Type":"ContainerStarted","Data":"bbe1cfe216478d8284bec526b8fe0c8b4ec5e1525b4114807982363831293a6d"} Jul 06 21:07:51.719558 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:51.719491 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-bfg7b" event={"ID":"c1601a80-baef-418f-9118-c5a7a0451d71","Type":"ContainerStarted","Data":"96e0cc32a9f9e831be0012484dd07c6ca801ebc781e39b39bb0a23d5c7b6faa1"} Jul 06 21:07:51.719718 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:51.719574 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-bfg7b" event={"ID":"c1601a80-baef-418f-9118-c5a7a0451d71","Type":"ContainerStarted","Data":"d8f09e88a2f5097c2b7e8e1c6d09d6d1b6f3ee1885245b279b2befccdb42d802"} Jul 06 21:07:51.719718 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:51.719579 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-bfg7b" podUID="c1601a80-baef-418f-9118-c5a7a0451d71" containerName="authorino" containerID="cri-o://96e0cc32a9f9e831be0012484dd07c6ca801ebc781e39b39bb0a23d5c7b6faa1" gracePeriod=30 Jul 06 21:07:51.750853 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:51.750782 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-66569b5d8f-8kzcm" podStartSLOduration=1.206480998 podStartE2EDuration="1.750755952s" totalImagesPullingTime="544.274954ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:07:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:07:51.746138517 +0000 UTC m=+556.064361781" watchObservedRunningTime="2026-07-06 21:07:51.750755952 +0000 UTC m=+556.068979213" Jul 06 21:07:51.767825 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:51.767766 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-bfg7b" podStartSLOduration=1.4008669679999999 podStartE2EDuration="1.76774776s" totalImagesPullingTime="366.880792ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:07:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:07:51.76668098 +0000 UTC m=+556.084904236" watchObservedRunningTime="2026-07-06 21:07:51.76774776 +0000 UTC m=+556.085971019" Jul 06 21:07:51.786505 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:51.786454 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-96rps"] Jul 06 21:07:51.786705 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:51.786678 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-96rps" podUID="37938462-8eef-4675-b24c-9e6433fcf740" containerName="authorino" containerID="cri-o://ef4ca5d8ba094d9875ae6425e99381aae40756e4b4a7d3cc5c002c40fc21b049" gracePeriod=30 Jul 06 21:07:52.037239 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.037203 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-bfg7b" Jul 06 21:07:52.066356 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.066326 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-96rps" Jul 06 21:07:52.078858 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.078637 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jb482\" (UniqueName: \"kubernetes.io/projected/c1601a80-baef-418f-9118-c5a7a0451d71-kube-api-access-jb482\") pod \"c1601a80-baef-418f-9118-c5a7a0451d71\" (UID: \"c1601a80-baef-418f-9118-c5a7a0451d71\") " Jul 06 21:07:52.078858 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.078782 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wfmr\" (UniqueName: \"kubernetes.io/projected/37938462-8eef-4675-b24c-9e6433fcf740-kube-api-access-7wfmr\") pod \"37938462-8eef-4675-b24c-9e6433fcf740\" (UID: \"37938462-8eef-4675-b24c-9e6433fcf740\") " Jul 06 21:07:52.081041 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.080990 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1601a80-baef-418f-9118-c5a7a0451d71-kube-api-access-jb482" (OuterVolumeSpecName: "kube-api-access-jb482") pod "c1601a80-baef-418f-9118-c5a7a0451d71" (UID: "c1601a80-baef-418f-9118-c5a7a0451d71"). InnerVolumeSpecName "kube-api-access-jb482". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:07:52.081167 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.081068 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37938462-8eef-4675-b24c-9e6433fcf740-kube-api-access-7wfmr" (OuterVolumeSpecName: "kube-api-access-7wfmr") pod "37938462-8eef-4675-b24c-9e6433fcf740" (UID: "37938462-8eef-4675-b24c-9e6433fcf740"). InnerVolumeSpecName "kube-api-access-7wfmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:07:52.131055 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.131000 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5b8ddb49c7-t2682"] Jul 06 21:07:52.131413 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.131400 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37938462-8eef-4675-b24c-9e6433fcf740" containerName="authorino" Jul 06 21:07:52.131462 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.131415 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="37938462-8eef-4675-b24c-9e6433fcf740" containerName="authorino" Jul 06 21:07:52.131462 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.131430 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1601a80-baef-418f-9118-c5a7a0451d71" containerName="authorino" Jul 06 21:07:52.131462 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.131435 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1601a80-baef-418f-9118-c5a7a0451d71" containerName="authorino" Jul 06 21:07:52.131558 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.131493 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1601a80-baef-418f-9118-c5a7a0451d71" containerName="authorino" Jul 06 21:07:52.131558 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.131502 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="37938462-8eef-4675-b24c-9e6433fcf740" containerName="authorino" Jul 06 21:07:52.134445 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.134415 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b8ddb49c7-t2682" Jul 06 21:07:52.138868 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.138838 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 06 21:07:52.162559 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.162465 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b8ddb49c7-t2682"] Jul 06 21:07:52.179597 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.179571 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt58v\" (UniqueName: \"kubernetes.io/projected/211614d0-f797-46df-9888-2cd5b66e0c45-kube-api-access-bt58v\") pod \"authorino-5b8ddb49c7-t2682\" (UID: \"211614d0-f797-46df-9888-2cd5b66e0c45\") " pod="kuadrant-system/authorino-5b8ddb49c7-t2682" Jul 06 21:07:52.179718 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.179638 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/211614d0-f797-46df-9888-2cd5b66e0c45-tls-cert\") pod \"authorino-5b8ddb49c7-t2682\" (UID: \"211614d0-f797-46df-9888-2cd5b66e0c45\") " pod="kuadrant-system/authorino-5b8ddb49c7-t2682" Jul 06 21:07:52.179773 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.179763 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wfmr\" (UniqueName: \"kubernetes.io/projected/37938462-8eef-4675-b24c-9e6433fcf740-kube-api-access-7wfmr\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:07:52.179811 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.179783 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb482\" (UniqueName: \"kubernetes.io/projected/c1601a80-baef-418f-9118-c5a7a0451d71-kube-api-access-jb482\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:07:52.280410 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.280365 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt58v\" (UniqueName: \"kubernetes.io/projected/211614d0-f797-46df-9888-2cd5b66e0c45-kube-api-access-bt58v\") pod \"authorino-5b8ddb49c7-t2682\" (UID: \"211614d0-f797-46df-9888-2cd5b66e0c45\") " pod="kuadrant-system/authorino-5b8ddb49c7-t2682" Jul 06 21:07:52.280586 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.280422 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/211614d0-f797-46df-9888-2cd5b66e0c45-tls-cert\") pod \"authorino-5b8ddb49c7-t2682\" (UID: \"211614d0-f797-46df-9888-2cd5b66e0c45\") " pod="kuadrant-system/authorino-5b8ddb49c7-t2682" Jul 06 21:07:52.282990 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.282955 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/211614d0-f797-46df-9888-2cd5b66e0c45-tls-cert\") pod \"authorino-5b8ddb49c7-t2682\" (UID: \"211614d0-f797-46df-9888-2cd5b66e0c45\") " pod="kuadrant-system/authorino-5b8ddb49c7-t2682" Jul 06 21:07:52.290214 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.290184 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt58v\" (UniqueName: \"kubernetes.io/projected/211614d0-f797-46df-9888-2cd5b66e0c45-kube-api-access-bt58v\") pod \"authorino-5b8ddb49c7-t2682\" (UID: \"211614d0-f797-46df-9888-2cd5b66e0c45\") " pod="kuadrant-system/authorino-5b8ddb49c7-t2682" Jul 06 21:07:52.445246 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.445156 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b8ddb49c7-t2682" Jul 06 21:07:52.609708 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.609624 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b8ddb49c7-t2682"] Jul 06 21:07:52.612302 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:07:52.612273 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod211614d0_f797_46df_9888_2cd5b66e0c45.slice/crio-ce31dd8eca1b073d49ba8f028a14c2bc5a215bfafe19c5213297390e1345cb88 WatchSource:0}: Error finding container ce31dd8eca1b073d49ba8f028a14c2bc5a215bfafe19c5213297390e1345cb88: Status 404 returned error can't find the container with id ce31dd8eca1b073d49ba8f028a14c2bc5a215bfafe19c5213297390e1345cb88 Jul 06 21:07:52.726758 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.726659 2577 generic.go:356] "Generic (PLEG): container finished" podID="c1601a80-baef-418f-9118-c5a7a0451d71" containerID="96e0cc32a9f9e831be0012484dd07c6ca801ebc781e39b39bb0a23d5c7b6faa1" exitCode=0 Jul 06 21:07:52.726758 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.726713 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-bfg7b" Jul 06 21:07:52.727339 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.726754 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-bfg7b" event={"ID":"c1601a80-baef-418f-9118-c5a7a0451d71","Type":"ContainerDied","Data":"96e0cc32a9f9e831be0012484dd07c6ca801ebc781e39b39bb0a23d5c7b6faa1"} Jul 06 21:07:52.727339 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.726794 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-bfg7b" event={"ID":"c1601a80-baef-418f-9118-c5a7a0451d71","Type":"ContainerDied","Data":"d8f09e88a2f5097c2b7e8e1c6d09d6d1b6f3ee1885245b279b2befccdb42d802"} Jul 06 21:07:52.727339 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.726819 2577 scope.go:117] "RemoveContainer" containerID="96e0cc32a9f9e831be0012484dd07c6ca801ebc781e39b39bb0a23d5c7b6faa1" Jul 06 21:07:52.728302 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.728275 2577 generic.go:356] "Generic (PLEG): container finished" podID="37938462-8eef-4675-b24c-9e6433fcf740" containerID="ef4ca5d8ba094d9875ae6425e99381aae40756e4b4a7d3cc5c002c40fc21b049" exitCode=0 Jul 06 21:07:52.728392 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.728330 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-96rps" Jul 06 21:07:52.728440 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.728403 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-96rps" event={"ID":"37938462-8eef-4675-b24c-9e6433fcf740","Type":"ContainerDied","Data":"ef4ca5d8ba094d9875ae6425e99381aae40756e4b4a7d3cc5c002c40fc21b049"} Jul 06 21:07:52.728485 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.728453 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-96rps" event={"ID":"37938462-8eef-4675-b24c-9e6433fcf740","Type":"ContainerDied","Data":"d06815c73e602fb84d9a90ef3ad8038aeb923c35d79dcbddd7faa288f99337b1"} Jul 06 21:07:52.730058 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.730028 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b8ddb49c7-t2682" event={"ID":"211614d0-f797-46df-9888-2cd5b66e0c45","Type":"ContainerStarted","Data":"ce31dd8eca1b073d49ba8f028a14c2bc5a215bfafe19c5213297390e1345cb88"} Jul 06 21:07:52.735960 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.735937 2577 scope.go:117] "RemoveContainer" containerID="96e0cc32a9f9e831be0012484dd07c6ca801ebc781e39b39bb0a23d5c7b6faa1" Jul 06 21:07:52.736338 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:07:52.736304 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96e0cc32a9f9e831be0012484dd07c6ca801ebc781e39b39bb0a23d5c7b6faa1\": container with ID starting with 96e0cc32a9f9e831be0012484dd07c6ca801ebc781e39b39bb0a23d5c7b6faa1 not found: ID does not exist" containerID="96e0cc32a9f9e831be0012484dd07c6ca801ebc781e39b39bb0a23d5c7b6faa1" Jul 06 21:07:52.736436 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.736347 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96e0cc32a9f9e831be0012484dd07c6ca801ebc781e39b39bb0a23d5c7b6faa1"} err="failed to get container status \"96e0cc32a9f9e831be0012484dd07c6ca801ebc781e39b39bb0a23d5c7b6faa1\": rpc error: code = NotFound desc = could not find container \"96e0cc32a9f9e831be0012484dd07c6ca801ebc781e39b39bb0a23d5c7b6faa1\": container with ID starting with 96e0cc32a9f9e831be0012484dd07c6ca801ebc781e39b39bb0a23d5c7b6faa1 not found: ID does not exist" Jul 06 21:07:52.736436 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.736377 2577 scope.go:117] "RemoveContainer" containerID="ef4ca5d8ba094d9875ae6425e99381aae40756e4b4a7d3cc5c002c40fc21b049" Jul 06 21:07:52.744821 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.744791 2577 scope.go:117] "RemoveContainer" containerID="ef4ca5d8ba094d9875ae6425e99381aae40756e4b4a7d3cc5c002c40fc21b049" Jul 06 21:07:52.745228 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:07:52.745206 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef4ca5d8ba094d9875ae6425e99381aae40756e4b4a7d3cc5c002c40fc21b049\": container with ID starting with ef4ca5d8ba094d9875ae6425e99381aae40756e4b4a7d3cc5c002c40fc21b049 not found: ID does not exist" containerID="ef4ca5d8ba094d9875ae6425e99381aae40756e4b4a7d3cc5c002c40fc21b049" Jul 06 21:07:52.745331 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.745237 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef4ca5d8ba094d9875ae6425e99381aae40756e4b4a7d3cc5c002c40fc21b049"} err="failed to get container status \"ef4ca5d8ba094d9875ae6425e99381aae40756e4b4a7d3cc5c002c40fc21b049\": rpc error: code = NotFound desc = could not find container \"ef4ca5d8ba094d9875ae6425e99381aae40756e4b4a7d3cc5c002c40fc21b049\": container with ID starting with ef4ca5d8ba094d9875ae6425e99381aae40756e4b4a7d3cc5c002c40fc21b049 not found: ID does not exist" Jul 06 21:07:52.757759 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.757717 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-96rps"] Jul 06 21:07:52.762066 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.762033 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-96rps"] Jul 06 21:07:52.788288 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.788251 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bfg7b"] Jul 06 21:07:52.792107 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:52.792077 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bfg7b"] Jul 06 21:07:53.738489 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:53.738448 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b8ddb49c7-t2682" event={"ID":"211614d0-f797-46df-9888-2cd5b66e0c45","Type":"ContainerStarted","Data":"15681dca261b8136dc9d416a849ac4e9062771123ff29433fffedc22f47f8038"} Jul 06 21:07:53.759822 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:53.759757 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5b8ddb49c7-t2682" podStartSLOduration=1.436308161 podStartE2EDuration="1.759736699s" totalImagesPullingTime="323.428538ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:07:53.75782274 +0000 UTC m=+558.076046001" watchObservedRunningTime="2026-07-06 21:07:53.759736699 +0000 UTC m=+558.077959960" Jul 06 21:07:53.794415 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:53.794382 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-66569b5d8f-8kzcm"] Jul 06 21:07:53.794591 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:53.794546 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-66569b5d8f-8kzcm" podUID="2d1e7bd3-8de7-4f61-a0b3-1619561f8b02" containerName="authorino" containerID="cri-o://195b5708b0404ece2bf77035dbfb9e35fcabc132ffd6e2bdc8a2adb07ff95951" gracePeriod=30 Jul 06 21:07:54.037612 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:54.037576 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66569b5d8f-8kzcm" Jul 06 21:07:54.095647 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:54.095606 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kc6j5\" (UniqueName: \"kubernetes.io/projected/2d1e7bd3-8de7-4f61-a0b3-1619561f8b02-kube-api-access-kc6j5\") pod \"2d1e7bd3-8de7-4f61-a0b3-1619561f8b02\" (UID: \"2d1e7bd3-8de7-4f61-a0b3-1619561f8b02\") " Jul 06 21:07:54.097784 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:54.097752 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d1e7bd3-8de7-4f61-a0b3-1619561f8b02-kube-api-access-kc6j5" (OuterVolumeSpecName: "kube-api-access-kc6j5") pod "2d1e7bd3-8de7-4f61-a0b3-1619561f8b02" (UID: "2d1e7bd3-8de7-4f61-a0b3-1619561f8b02"). InnerVolumeSpecName "kube-api-access-kc6j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:07:54.168912 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:54.168876 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37938462-8eef-4675-b24c-9e6433fcf740" path="/var/lib/kubelet/pods/37938462-8eef-4675-b24c-9e6433fcf740/volumes" Jul 06 21:07:54.169398 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:54.169363 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1601a80-baef-418f-9118-c5a7a0451d71" path="/var/lib/kubelet/pods/c1601a80-baef-418f-9118-c5a7a0451d71/volumes" Jul 06 21:07:54.200078 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:54.197188 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc6j5\" (UniqueName: \"kubernetes.io/projected/2d1e7bd3-8de7-4f61-a0b3-1619561f8b02-kube-api-access-kc6j5\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:07:54.744156 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:54.744117 2577 generic.go:356] "Generic (PLEG): container finished" podID="2d1e7bd3-8de7-4f61-a0b3-1619561f8b02" containerID="195b5708b0404ece2bf77035dbfb9e35fcabc132ffd6e2bdc8a2adb07ff95951" exitCode=0 Jul 06 21:07:54.744649 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:54.744182 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66569b5d8f-8kzcm" Jul 06 21:07:54.744649 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:54.744202 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66569b5d8f-8kzcm" event={"ID":"2d1e7bd3-8de7-4f61-a0b3-1619561f8b02","Type":"ContainerDied","Data":"195b5708b0404ece2bf77035dbfb9e35fcabc132ffd6e2bdc8a2adb07ff95951"} Jul 06 21:07:54.744649 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:54.744259 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66569b5d8f-8kzcm" event={"ID":"2d1e7bd3-8de7-4f61-a0b3-1619561f8b02","Type":"ContainerDied","Data":"bbe1cfe216478d8284bec526b8fe0c8b4ec5e1525b4114807982363831293a6d"} Jul 06 21:07:54.744649 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:54.744283 2577 scope.go:117] "RemoveContainer" containerID="195b5708b0404ece2bf77035dbfb9e35fcabc132ffd6e2bdc8a2adb07ff95951" Jul 06 21:07:54.754913 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:54.754856 2577 scope.go:117] "RemoveContainer" containerID="195b5708b0404ece2bf77035dbfb9e35fcabc132ffd6e2bdc8a2adb07ff95951" Jul 06 21:07:54.755601 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:07:54.755574 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"195b5708b0404ece2bf77035dbfb9e35fcabc132ffd6e2bdc8a2adb07ff95951\": container with ID starting with 195b5708b0404ece2bf77035dbfb9e35fcabc132ffd6e2bdc8a2adb07ff95951 not found: ID does not exist" containerID="195b5708b0404ece2bf77035dbfb9e35fcabc132ffd6e2bdc8a2adb07ff95951" Jul 06 21:07:54.755657 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:54.755612 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195b5708b0404ece2bf77035dbfb9e35fcabc132ffd6e2bdc8a2adb07ff95951"} err="failed to get container status \"195b5708b0404ece2bf77035dbfb9e35fcabc132ffd6e2bdc8a2adb07ff95951\": rpc error: code = NotFound desc = could not find container \"195b5708b0404ece2bf77035dbfb9e35fcabc132ffd6e2bdc8a2adb07ff95951\": container with ID starting with 195b5708b0404ece2bf77035dbfb9e35fcabc132ffd6e2bdc8a2adb07ff95951 not found: ID does not exist" Jul 06 21:07:54.763924 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:54.763891 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-66569b5d8f-8kzcm"] Jul 06 21:07:54.766632 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:54.766604 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-66569b5d8f-8kzcm"] Jul 06 21:07:56.169250 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:56.169217 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d1e7bd3-8de7-4f61-a0b3-1619561f8b02" path="/var/lib/kubelet/pods/2d1e7bd3-8de7-4f61-a0b3-1619561f8b02/volumes" Jul 06 21:07:56.353438 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:56.353403 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58c8f6c4f-qvd6w"] Jul 06 21:07:56.353795 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:56.353780 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d1e7bd3-8de7-4f61-a0b3-1619561f8b02" containerName="authorino" Jul 06 21:07:56.353795 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:56.353796 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d1e7bd3-8de7-4f61-a0b3-1619561f8b02" containerName="authorino" Jul 06 21:07:56.353902 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:56.353860 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d1e7bd3-8de7-4f61-a0b3-1619561f8b02" containerName="authorino" Jul 06 21:07:56.398138 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:56.398094 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58c8f6c4f-qvd6w"] Jul 06 21:07:56.398399 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:56.398210 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58c8f6c4f-qvd6w" Jul 06 21:07:56.401397 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:56.401253 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5cnb8\"" Jul 06 21:07:56.401397 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:56.401383 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 21:07:56.401590 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:56.401437 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 21:07:56.419414 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:56.419325 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkmzr\" (UniqueName: \"kubernetes.io/projected/a479cbdb-72b7-461b-acdb-dd3bf599983b-kube-api-access-tkmzr\") pod \"maas-controller-58c8f6c4f-qvd6w\" (UID: \"a479cbdb-72b7-461b-acdb-dd3bf599983b\") " pod="opendatahub/maas-controller-58c8f6c4f-qvd6w" Jul 06 21:07:56.419414 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:56.419407 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a479cbdb-72b7-461b-acdb-dd3bf599983b-webhook-cert\") pod \"maas-controller-58c8f6c4f-qvd6w\" (UID: \"a479cbdb-72b7-461b-acdb-dd3bf599983b\") " pod="opendatahub/maas-controller-58c8f6c4f-qvd6w" Jul 06 21:07:56.520312 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:56.520270 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkmzr\" (UniqueName: \"kubernetes.io/projected/a479cbdb-72b7-461b-acdb-dd3bf599983b-kube-api-access-tkmzr\") pod \"maas-controller-58c8f6c4f-qvd6w\" (UID: \"a479cbdb-72b7-461b-acdb-dd3bf599983b\") " pod="opendatahub/maas-controller-58c8f6c4f-qvd6w" Jul 06 21:07:56.520542 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:56.520414 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a479cbdb-72b7-461b-acdb-dd3bf599983b-webhook-cert\") pod \"maas-controller-58c8f6c4f-qvd6w\" (UID: \"a479cbdb-72b7-461b-acdb-dd3bf599983b\") " pod="opendatahub/maas-controller-58c8f6c4f-qvd6w" Jul 06 21:07:56.522899 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:56.522872 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a479cbdb-72b7-461b-acdb-dd3bf599983b-webhook-cert\") pod \"maas-controller-58c8f6c4f-qvd6w\" (UID: \"a479cbdb-72b7-461b-acdb-dd3bf599983b\") " pod="opendatahub/maas-controller-58c8f6c4f-qvd6w" Jul 06 21:07:56.530296 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:56.530270 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkmzr\" (UniqueName: \"kubernetes.io/projected/a479cbdb-72b7-461b-acdb-dd3bf599983b-kube-api-access-tkmzr\") pod \"maas-controller-58c8f6c4f-qvd6w\" (UID: \"a479cbdb-72b7-461b-acdb-dd3bf599983b\") " pod="opendatahub/maas-controller-58c8f6c4f-qvd6w" Jul 06 21:07:56.549344 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:56.549307 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-67f746bd7b-b2kkn"] Jul 06 21:07:56.566546 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:56.566509 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67f746bd7b-b2kkn"] Jul 06 21:07:56.566726 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:56.566635 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67f746bd7b-b2kkn" Jul 06 21:07:56.621570 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:56.621534 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e7b76dc-5003-4a19-a18d-7d3c6c375be1-webhook-cert\") pod \"maas-controller-67f746bd7b-b2kkn\" (UID: \"2e7b76dc-5003-4a19-a18d-7d3c6c375be1\") " pod="opendatahub/maas-controller-67f746bd7b-b2kkn" Jul 06 21:07:56.621751 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:56.621649 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kz8f\" (UniqueName: \"kubernetes.io/projected/2e7b76dc-5003-4a19-a18d-7d3c6c375be1-kube-api-access-9kz8f\") pod \"maas-controller-67f746bd7b-b2kkn\" (UID: \"2e7b76dc-5003-4a19-a18d-7d3c6c375be1\") " pod="opendatahub/maas-controller-67f746bd7b-b2kkn" Jul 06 21:07:56.712175 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:56.712080 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58c8f6c4f-qvd6w" Jul 06 21:07:56.722161 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:56.722138 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kz8f\" (UniqueName: \"kubernetes.io/projected/2e7b76dc-5003-4a19-a18d-7d3c6c375be1-kube-api-access-9kz8f\") pod \"maas-controller-67f746bd7b-b2kkn\" (UID: \"2e7b76dc-5003-4a19-a18d-7d3c6c375be1\") " pod="opendatahub/maas-controller-67f746bd7b-b2kkn" Jul 06 21:07:56.722263 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:56.722192 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e7b76dc-5003-4a19-a18d-7d3c6c375be1-webhook-cert\") pod \"maas-controller-67f746bd7b-b2kkn\" (UID: \"2e7b76dc-5003-4a19-a18d-7d3c6c375be1\") " pod="opendatahub/maas-controller-67f746bd7b-b2kkn" Jul 06 21:07:56.724896 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:56.724862 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e7b76dc-5003-4a19-a18d-7d3c6c375be1-webhook-cert\") pod \"maas-controller-67f746bd7b-b2kkn\" (UID: \"2e7b76dc-5003-4a19-a18d-7d3c6c375be1\") " pod="opendatahub/maas-controller-67f746bd7b-b2kkn" Jul 06 21:07:56.732787 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:56.732755 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kz8f\" (UniqueName: \"kubernetes.io/projected/2e7b76dc-5003-4a19-a18d-7d3c6c375be1-kube-api-access-9kz8f\") pod \"maas-controller-67f746bd7b-b2kkn\" (UID: \"2e7b76dc-5003-4a19-a18d-7d3c6c375be1\") " pod="opendatahub/maas-controller-67f746bd7b-b2kkn" Jul 06 21:07:56.854576 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:56.854502 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58c8f6c4f-qvd6w"] Jul 06 21:07:56.856983 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:07:56.856953 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda479cbdb_72b7_461b_acdb_dd3bf599983b.slice/crio-7c82eb6472b1bb01299d4d5261f6208c0cf5b398aad2459629ec9d71a6fa0b2b WatchSource:0}: Error finding container 7c82eb6472b1bb01299d4d5261f6208c0cf5b398aad2459629ec9d71a6fa0b2b: Status 404 returned error can't find the container with id 7c82eb6472b1bb01299d4d5261f6208c0cf5b398aad2459629ec9d71a6fa0b2b Jul 06 21:07:56.878185 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:56.878142 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67f746bd7b-b2kkn" Jul 06 21:07:57.021934 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:57.021909 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67f746bd7b-b2kkn"] Jul 06 21:07:57.023810 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:07:57.023779 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e7b76dc_5003_4a19_a18d_7d3c6c375be1.slice/crio-dfe76a703b13d0c21fe0f55b3df638f5323a794ff30fd1dcdba421d21248aadb WatchSource:0}: Error finding container dfe76a703b13d0c21fe0f55b3df638f5323a794ff30fd1dcdba421d21248aadb: Status 404 returned error can't find the container with id dfe76a703b13d0c21fe0f55b3df638f5323a794ff30fd1dcdba421d21248aadb Jul 06 21:07:57.766264 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:57.766214 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67f746bd7b-b2kkn" event={"ID":"2e7b76dc-5003-4a19-a18d-7d3c6c375be1","Type":"ContainerStarted","Data":"dfe76a703b13d0c21fe0f55b3df638f5323a794ff30fd1dcdba421d21248aadb"} Jul 06 21:07:57.769256 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:57.769218 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58c8f6c4f-qvd6w" event={"ID":"a479cbdb-72b7-461b-acdb-dd3bf599983b","Type":"ContainerStarted","Data":"7c82eb6472b1bb01299d4d5261f6208c0cf5b398aad2459629ec9d71a6fa0b2b"} Jul 06 21:07:59.781061 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:59.781001 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67f746bd7b-b2kkn" event={"ID":"2e7b76dc-5003-4a19-a18d-7d3c6c375be1","Type":"ContainerStarted","Data":"6a839025ec504bc0e3f7bf50c55fd3d306f54d0801c24d8c339e0e6c63bdf675"} Jul 06 21:07:59.781522 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:59.781088 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-67f746bd7b-b2kkn" Jul 06 21:07:59.782507 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:59.782482 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58c8f6c4f-qvd6w" event={"ID":"a479cbdb-72b7-461b-acdb-dd3bf599983b","Type":"ContainerStarted","Data":"79e0d9ea9f8aaf2fa8450cb65e104a356ada124162310483235ab6ccb442434d"} Jul 06 21:07:59.782622 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:59.782606 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58c8f6c4f-qvd6w" Jul 06 21:07:59.812617 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:59.812560 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-67f746bd7b-b2kkn" podStartSLOduration=1.706598272 podStartE2EDuration="3.812544072s" totalImagesPullingTime="2.1059458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:07:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:07:59.809071202 +0000 UTC m=+564.127294474" watchObservedRunningTime="2026-07-06 21:07:59.812544072 +0000 UTC m=+564.130767310" Jul 06 21:07:59.848319 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:07:59.848254 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58c8f6c4f-qvd6w" podStartSLOduration=1.5648698699999999 podStartE2EDuration="3.848233074s" totalImagesPullingTime="2.283363204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:07:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:07:59.845679902 +0000 UTC m=+564.163903162" watchObservedRunningTime="2026-07-06 21:07:59.848233074 +0000 UTC m=+564.166456333" Jul 06 21:08:10.793172 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:10.793141 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58c8f6c4f-qvd6w" Jul 06 21:08:10.793593 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:10.793220 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-67f746bd7b-b2kkn" Jul 06 21:08:10.863553 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:10.863518 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58c8f6c4f-qvd6w"] Jul 06 21:08:10.863973 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:10.863934 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-58c8f6c4f-qvd6w" podUID="a479cbdb-72b7-461b-acdb-dd3bf599983b" containerName="manager" containerID="cri-o://79e0d9ea9f8aaf2fa8450cb65e104a356ada124162310483235ab6ccb442434d" gracePeriod=30 Jul 06 21:08:11.115427 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:11.115401 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58c8f6c4f-qvd6w" Jul 06 21:08:11.261767 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:11.261733 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkmzr\" (UniqueName: \"kubernetes.io/projected/a479cbdb-72b7-461b-acdb-dd3bf599983b-kube-api-access-tkmzr\") pod \"a479cbdb-72b7-461b-acdb-dd3bf599983b\" (UID: \"a479cbdb-72b7-461b-acdb-dd3bf599983b\") " Jul 06 21:08:11.261937 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:11.261854 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a479cbdb-72b7-461b-acdb-dd3bf599983b-webhook-cert\") pod \"a479cbdb-72b7-461b-acdb-dd3bf599983b\" (UID: \"a479cbdb-72b7-461b-acdb-dd3bf599983b\") " Jul 06 21:08:11.264039 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:11.263989 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a479cbdb-72b7-461b-acdb-dd3bf599983b-kube-api-access-tkmzr" (OuterVolumeSpecName: "kube-api-access-tkmzr") pod "a479cbdb-72b7-461b-acdb-dd3bf599983b" (UID: "a479cbdb-72b7-461b-acdb-dd3bf599983b"). InnerVolumeSpecName "kube-api-access-tkmzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:08:11.264039 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:11.263987 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a479cbdb-72b7-461b-acdb-dd3bf599983b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "a479cbdb-72b7-461b-acdb-dd3bf599983b" (UID: "a479cbdb-72b7-461b-acdb-dd3bf599983b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:08:11.363514 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:11.363417 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkmzr\" (UniqueName: \"kubernetes.io/projected/a479cbdb-72b7-461b-acdb-dd3bf599983b-kube-api-access-tkmzr\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:08:11.363514 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:11.363453 2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a479cbdb-72b7-461b-acdb-dd3bf599983b-webhook-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:08:11.846275 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:11.846229 2577 generic.go:356] "Generic (PLEG): container finished" podID="a479cbdb-72b7-461b-acdb-dd3bf599983b" containerID="79e0d9ea9f8aaf2fa8450cb65e104a356ada124162310483235ab6ccb442434d" exitCode=0 Jul 06 21:08:11.846275 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:11.846284 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58c8f6c4f-qvd6w" Jul 06 21:08:11.846813 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:11.846316 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58c8f6c4f-qvd6w" event={"ID":"a479cbdb-72b7-461b-acdb-dd3bf599983b","Type":"ContainerDied","Data":"79e0d9ea9f8aaf2fa8450cb65e104a356ada124162310483235ab6ccb442434d"} Jul 06 21:08:11.846813 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:11.846379 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58c8f6c4f-qvd6w" event={"ID":"a479cbdb-72b7-461b-acdb-dd3bf599983b","Type":"ContainerDied","Data":"7c82eb6472b1bb01299d4d5261f6208c0cf5b398aad2459629ec9d71a6fa0b2b"} Jul 06 21:08:11.846813 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:11.846411 2577 scope.go:117] "RemoveContainer" containerID="79e0d9ea9f8aaf2fa8450cb65e104a356ada124162310483235ab6ccb442434d" Jul 06 21:08:11.856212 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:11.856193 2577 scope.go:117] "RemoveContainer" containerID="79e0d9ea9f8aaf2fa8450cb65e104a356ada124162310483235ab6ccb442434d" Jul 06 21:08:11.856530 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:08:11.856507 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79e0d9ea9f8aaf2fa8450cb65e104a356ada124162310483235ab6ccb442434d\": container with ID starting with 79e0d9ea9f8aaf2fa8450cb65e104a356ada124162310483235ab6ccb442434d not found: ID does not exist" containerID="79e0d9ea9f8aaf2fa8450cb65e104a356ada124162310483235ab6ccb442434d" Jul 06 21:08:11.856600 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:11.856543 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79e0d9ea9f8aaf2fa8450cb65e104a356ada124162310483235ab6ccb442434d"} err="failed to get container status \"79e0d9ea9f8aaf2fa8450cb65e104a356ada124162310483235ab6ccb442434d\": rpc error: code = NotFound desc = could not find container \"79e0d9ea9f8aaf2fa8450cb65e104a356ada124162310483235ab6ccb442434d\": container with ID starting with 79e0d9ea9f8aaf2fa8450cb65e104a356ada124162310483235ab6ccb442434d not found: ID does not exist" Jul 06 21:08:11.870354 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:11.870319 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58c8f6c4f-qvd6w"] Jul 06 21:08:11.876602 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:11.876564 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58c8f6c4f-qvd6w"] Jul 06 21:08:12.168783 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:12.168746 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a479cbdb-72b7-461b-acdb-dd3bf599983b" path="/var/lib/kubelet/pods/a479cbdb-72b7-461b-acdb-dd3bf599983b/volumes" Jul 06 21:08:34.269605 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.269567 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-688cc6cd9f-d78bw"] Jul 06 21:08:34.270111 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.269921 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a479cbdb-72b7-461b-acdb-dd3bf599983b" containerName="manager" Jul 06 21:08:34.270111 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.269932 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a479cbdb-72b7-461b-acdb-dd3bf599983b" containerName="manager" Jul 06 21:08:34.270111 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.270003 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a479cbdb-72b7-461b-acdb-dd3bf599983b" containerName="manager" Jul 06 21:08:34.272349 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.272330 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-688cc6cd9f-d78bw" Jul 06 21:08:34.275565 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.275539 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 06 21:08:34.284811 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.284785 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-nqlrr\"" Jul 06 21:08:34.288257 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.288230 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-688cc6cd9f-d78bw"] Jul 06 21:08:34.368390 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.368356 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7gsw\" (UniqueName: \"kubernetes.io/projected/013bc0d1-86de-4f5e-bce5-04c3e39798fa-kube-api-access-d7gsw\") pod \"maas-api-688cc6cd9f-d78bw\" (UID: \"013bc0d1-86de-4f5e-bce5-04c3e39798fa\") " pod="opendatahub/maas-api-688cc6cd9f-d78bw" Jul 06 21:08:34.368390 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.368403 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/013bc0d1-86de-4f5e-bce5-04c3e39798fa-openshift-service-ca\") pod \"maas-api-688cc6cd9f-d78bw\" (UID: \"013bc0d1-86de-4f5e-bce5-04c3e39798fa\") " pod="opendatahub/maas-api-688cc6cd9f-d78bw" Jul 06 21:08:34.368650 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.368514 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/013bc0d1-86de-4f5e-bce5-04c3e39798fa-maas-api-tls\") pod \"maas-api-688cc6cd9f-d78bw\" (UID: \"013bc0d1-86de-4f5e-bce5-04c3e39798fa\") " pod="opendatahub/maas-api-688cc6cd9f-d78bw" Jul 06 21:08:34.469183 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.469148 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7gsw\" (UniqueName: \"kubernetes.io/projected/013bc0d1-86de-4f5e-bce5-04c3e39798fa-kube-api-access-d7gsw\") pod \"maas-api-688cc6cd9f-d78bw\" (UID: \"013bc0d1-86de-4f5e-bce5-04c3e39798fa\") " pod="opendatahub/maas-api-688cc6cd9f-d78bw" Jul 06 21:08:34.469376 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.469196 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/013bc0d1-86de-4f5e-bce5-04c3e39798fa-openshift-service-ca\") pod \"maas-api-688cc6cd9f-d78bw\" (UID: \"013bc0d1-86de-4f5e-bce5-04c3e39798fa\") " pod="opendatahub/maas-api-688cc6cd9f-d78bw" Jul 06 21:08:34.469376 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.469232 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/013bc0d1-86de-4f5e-bce5-04c3e39798fa-maas-api-tls\") pod \"maas-api-688cc6cd9f-d78bw\" (UID: \"013bc0d1-86de-4f5e-bce5-04c3e39798fa\") " pod="opendatahub/maas-api-688cc6cd9f-d78bw" Jul 06 21:08:34.469914 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.469889 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/013bc0d1-86de-4f5e-bce5-04c3e39798fa-openshift-service-ca\") pod \"maas-api-688cc6cd9f-d78bw\" (UID: \"013bc0d1-86de-4f5e-bce5-04c3e39798fa\") " pod="opendatahub/maas-api-688cc6cd9f-d78bw" Jul 06 21:08:34.471965 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.471944 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/013bc0d1-86de-4f5e-bce5-04c3e39798fa-maas-api-tls\") pod \"maas-api-688cc6cd9f-d78bw\" (UID: \"013bc0d1-86de-4f5e-bce5-04c3e39798fa\") " pod="opendatahub/maas-api-688cc6cd9f-d78bw" Jul 06 21:08:34.478192 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.478168 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7gsw\" (UniqueName: \"kubernetes.io/projected/013bc0d1-86de-4f5e-bce5-04c3e39798fa-kube-api-access-d7gsw\") pod \"maas-api-688cc6cd9f-d78bw\" (UID: \"013bc0d1-86de-4f5e-bce5-04c3e39798fa\") " pod="opendatahub/maas-api-688cc6cd9f-d78bw" Jul 06 21:08:34.583452 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.583361 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-688cc6cd9f-d78bw" Jul 06 21:08:34.739289 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.739260 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-688cc6cd9f-d78bw"] Jul 06 21:08:34.741832 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:08:34.741806 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod013bc0d1_86de_4f5e_bce5_04c3e39798fa.slice/crio-17e3ba09e2d77585f3dc8619656f6173fe0cd56989f7ec4fe17f5e7b12d91ca5 WatchSource:0}: Error finding container 17e3ba09e2d77585f3dc8619656f6173fe0cd56989f7ec4fe17f5e7b12d91ca5: Status 404 returned error can't find the container with id 17e3ba09e2d77585f3dc8619656f6173fe0cd56989f7ec4fe17f5e7b12d91ca5 Jul 06 21:08:34.761930 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.761898 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-sk8mt"] Jul 06 21:08:34.765264 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.765244 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-sk8mt" Jul 06 21:08:34.767752 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.767726 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 06 21:08:34.767925 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.767785 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-hnjzr\"" Jul 06 21:08:34.771559 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.771527 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4dad928-112f-4baf-a79d-c9294ddadbd1-plugins-config-volume\") pod \"payload-processing-58f86c7798-sk8mt\" (UID: \"c4dad928-112f-4baf-a79d-c9294ddadbd1\") " pod="openshift-ingress/payload-processing-58f86c7798-sk8mt" Jul 06 21:08:34.771700 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.771579 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nb9v\" (UniqueName: \"kubernetes.io/projected/c4dad928-112f-4baf-a79d-c9294ddadbd1-kube-api-access-4nb9v\") pod \"payload-processing-58f86c7798-sk8mt\" (UID: \"c4dad928-112f-4baf-a79d-c9294ddadbd1\") " pod="openshift-ingress/payload-processing-58f86c7798-sk8mt" Jul 06 21:08:34.778624 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.778597 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-sk8mt"] Jul 06 21:08:34.830785 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.830746 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-nrqcv"] Jul 06 21:08:34.833537 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.833485 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-nrqcv" Jul 06 21:08:34.847464 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.847437 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-nrqcv"] Jul 06 21:08:34.872683 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.872650 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6c09920-f5fb-4db5-b5de-06d64bfb48f9-plugins-config-volume\") pod \"payload-pre-processing-7964457696-nrqcv\" (UID: \"e6c09920-f5fb-4db5-b5de-06d64bfb48f9\") " pod="openshift-ingress/payload-pre-processing-7964457696-nrqcv" Jul 06 21:08:34.872836 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.872701 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4dad928-112f-4baf-a79d-c9294ddadbd1-plugins-config-volume\") pod \"payload-processing-58f86c7798-sk8mt\" (UID: \"c4dad928-112f-4baf-a79d-c9294ddadbd1\") " pod="openshift-ingress/payload-processing-58f86c7798-sk8mt" Jul 06 21:08:34.872836 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.872719 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74ksn\" (UniqueName: \"kubernetes.io/projected/e6c09920-f5fb-4db5-b5de-06d64bfb48f9-kube-api-access-74ksn\") pod \"payload-pre-processing-7964457696-nrqcv\" (UID: \"e6c09920-f5fb-4db5-b5de-06d64bfb48f9\") " pod="openshift-ingress/payload-pre-processing-7964457696-nrqcv" Jul 06 21:08:34.872836 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.872761 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nb9v\" (UniqueName: \"kubernetes.io/projected/c4dad928-112f-4baf-a79d-c9294ddadbd1-kube-api-access-4nb9v\") pod \"payload-processing-58f86c7798-sk8mt\" (UID: \"c4dad928-112f-4baf-a79d-c9294ddadbd1\") " pod="openshift-ingress/payload-processing-58f86c7798-sk8mt" Jul 06 21:08:34.873471 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.873452 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4dad928-112f-4baf-a79d-c9294ddadbd1-plugins-config-volume\") pod \"payload-processing-58f86c7798-sk8mt\" (UID: \"c4dad928-112f-4baf-a79d-c9294ddadbd1\") " pod="openshift-ingress/payload-processing-58f86c7798-sk8mt" Jul 06 21:08:34.882483 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.882447 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nb9v\" (UniqueName: \"kubernetes.io/projected/c4dad928-112f-4baf-a79d-c9294ddadbd1-kube-api-access-4nb9v\") pod \"payload-processing-58f86c7798-sk8mt\" (UID: \"c4dad928-112f-4baf-a79d-c9294ddadbd1\") " pod="openshift-ingress/payload-processing-58f86c7798-sk8mt" Jul 06 21:08:34.976414 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.976365 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6c09920-f5fb-4db5-b5de-06d64bfb48f9-plugins-config-volume\") pod \"payload-pre-processing-7964457696-nrqcv\" (UID: \"e6c09920-f5fb-4db5-b5de-06d64bfb48f9\") " pod="openshift-ingress/payload-pre-processing-7964457696-nrqcv" Jul 06 21:08:34.976618 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.976465 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74ksn\" (UniqueName: \"kubernetes.io/projected/e6c09920-f5fb-4db5-b5de-06d64bfb48f9-kube-api-access-74ksn\") pod \"payload-pre-processing-7964457696-nrqcv\" (UID: \"e6c09920-f5fb-4db5-b5de-06d64bfb48f9\") " pod="openshift-ingress/payload-pre-processing-7964457696-nrqcv" Jul 06 21:08:34.977766 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.977489 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6c09920-f5fb-4db5-b5de-06d64bfb48f9-plugins-config-volume\") pod \"payload-pre-processing-7964457696-nrqcv\" (UID: \"e6c09920-f5fb-4db5-b5de-06d64bfb48f9\") " pod="openshift-ingress/payload-pre-processing-7964457696-nrqcv" Jul 06 21:08:34.984332 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.984282 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-688cc6cd9f-d78bw" event={"ID":"013bc0d1-86de-4f5e-bce5-04c3e39798fa","Type":"ContainerStarted","Data":"17e3ba09e2d77585f3dc8619656f6173fe0cd56989f7ec4fe17f5e7b12d91ca5"} Jul 06 21:08:34.987637 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:34.987598 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74ksn\" (UniqueName: \"kubernetes.io/projected/e6c09920-f5fb-4db5-b5de-06d64bfb48f9-kube-api-access-74ksn\") pod \"payload-pre-processing-7964457696-nrqcv\" (UID: \"e6c09920-f5fb-4db5-b5de-06d64bfb48f9\") " pod="openshift-ingress/payload-pre-processing-7964457696-nrqcv" Jul 06 21:08:35.076873 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:35.076826 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-sk8mt" Jul 06 21:08:35.144439 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:35.144405 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-nrqcv" Jul 06 21:08:35.225530 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:35.225500 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-sk8mt"] Jul 06 21:08:35.228265 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:08:35.228220 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4dad928_112f_4baf_a79d_c9294ddadbd1.slice/crio-ef38158cf9bd3b497478e77f419da986f76a4b739d8f98c3fed72783697fb273 WatchSource:0}: Error finding container ef38158cf9bd3b497478e77f419da986f76a4b739d8f98c3fed72783697fb273: Status 404 returned error can't find the container with id ef38158cf9bd3b497478e77f419da986f76a4b739d8f98c3fed72783697fb273 Jul 06 21:08:35.294373 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:35.294345 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-nrqcv"] Jul 06 21:08:35.296860 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:08:35.296827 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c09920_f5fb_4db5_b5de_06d64bfb48f9.slice/crio-32f68da4e09234c8f38acdcab20159b0ecf1076d8c2e37aeeacf9c978e125896 WatchSource:0}: Error finding container 32f68da4e09234c8f38acdcab20159b0ecf1076d8c2e37aeeacf9c978e125896: Status 404 returned error can't find the container with id 32f68da4e09234c8f38acdcab20159b0ecf1076d8c2e37aeeacf9c978e125896 Jul 06 21:08:35.993214 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:35.993135 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-nrqcv" event={"ID":"e6c09920-f5fb-4db5-b5de-06d64bfb48f9","Type":"ContainerStarted","Data":"32f68da4e09234c8f38acdcab20159b0ecf1076d8c2e37aeeacf9c978e125896"} Jul 06 21:08:35.994997 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:35.994919 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-sk8mt" event={"ID":"c4dad928-112f-4baf-a79d-c9294ddadbd1","Type":"ContainerStarted","Data":"ef38158cf9bd3b497478e77f419da986f76a4b739d8f98c3fed72783697fb273"} Jul 06 21:08:36.155914 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:36.155887 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qjs2n_64b45669-ed56-41b9-a3ed-1fe5c13508b6/ovn-acl-logging/0.log" Jul 06 21:08:36.160071 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:36.159984 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qjs2n_64b45669-ed56-41b9-a3ed-1fe5c13508b6/ovn-acl-logging/0.log" Jul 06 21:08:38.007162 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:38.007120 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-688cc6cd9f-d78bw" event={"ID":"013bc0d1-86de-4f5e-bce5-04c3e39798fa","Type":"ContainerStarted","Data":"aa4347ac74b0103a5c77fc5e34aec6ccc2adb006938f638a5117439dc977123f"} Jul 06 21:08:38.007794 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:38.007272 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-688cc6cd9f-d78bw" Jul 06 21:08:38.008624 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:38.008601 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-nrqcv" event={"ID":"e6c09920-f5fb-4db5-b5de-06d64bfb48f9","Type":"ContainerStarted","Data":"942901152eae94314ad474df5e2215d7df19b0f63c4f7e1dbae88395037bd9d6"} Jul 06 21:08:38.008710 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:38.008652 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-nrqcv" Jul 06 21:08:38.010051 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:38.010029 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-sk8mt" event={"ID":"c4dad928-112f-4baf-a79d-c9294ddadbd1","Type":"ContainerStarted","Data":"f1f69c93ec97db43987af080e1c707f140039ca064b8e5214560809482e373eb"} Jul 06 21:08:38.010171 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:38.010077 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-sk8mt" Jul 06 21:08:38.029053 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:38.028968 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-688cc6cd9f-d78bw" podStartSLOduration=1.470409439 podStartE2EDuration="4.02894421s" totalImagesPullingTime="2.558534771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:08:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:08:38.02694427 +0000 UTC m=+602.345167554" watchObservedRunningTime="2026-07-06 21:08:38.02894421 +0000 UTC m=+602.347167471" Jul 06 21:08:38.056659 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:38.056576 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-nrqcv" podStartSLOduration=2.053787535 podStartE2EDuration="4.05654729s" totalImagesPullingTime="2.002759755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:08:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:08:38.044471933 +0000 UTC m=+602.362695191" watchObservedRunningTime="2026-07-06 21:08:38.05654729 +0000 UTC m=+602.374770551" Jul 06 21:08:38.066999 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:38.066899 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-sk8mt" podStartSLOduration=1.993168057 podStartE2EDuration="4.066850439s" totalImagesPullingTime="2.073682382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:08:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:08:38.065462212 +0000 UTC m=+602.383685474" watchObservedRunningTime="2026-07-06 21:08:38.066850439 +0000 UTC m=+602.385073690" Jul 06 21:08:43.036297 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:43.036259 2577 generic.go:356] "Generic (PLEG): container finished" podID="90be9695-5aa7-4767-95a3-81acb10d1237" containerID="c29cd1a039bc0cb6923a0c91846eaca929815e5e85a3ac03e4f21c1ed8480fdd" exitCode=0 Jul 06 21:08:43.036711 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:43.036335 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" event={"ID":"90be9695-5aa7-4767-95a3-81acb10d1237","Type":"ContainerDied","Data":"c29cd1a039bc0cb6923a0c91846eaca929815e5e85a3ac03e4f21c1ed8480fdd"} Jul 06 21:08:43.036753 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:43.036743 2577 scope.go:117] "RemoveContainer" containerID="c29cd1a039bc0cb6923a0c91846eaca929815e5e85a3ac03e4f21c1ed8480fdd" Jul 06 21:08:43.037767 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:43.037742 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:08:43.037846 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:43.037830 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:08:43.037893 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:43.037878 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:08:44.020631 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:44.020597 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-688cc6cd9f-d78bw" Jul 06 21:08:44.046549 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:44.046513 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" event={"ID":"90be9695-5aa7-4767-95a3-81acb10d1237","Type":"ContainerStarted","Data":"aca69b5c29e9e31a2135cb0a3efcdfed6ac54ed80f5949c9227877cdf6c02878"} Jul 06 21:08:44.178803 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:44.178764 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:08:44.179943 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:44.179914 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-77zzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 06 21:08:44.180075 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:44.179982 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" podUID="90be9695-5aa7-4767-95a3-81acb10d1237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 06 21:08:44.443642 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:44.443604 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:08:45.179024 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:45.178976 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-77zzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 06 21:08:45.179433 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:45.179066 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" podUID="90be9695-5aa7-4767-95a3-81acb10d1237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 06 21:08:46.178264 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:46.178228 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-77zzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 06 21:08:46.178482 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:46.178293 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" podUID="90be9695-5aa7-4767-95a3-81acb10d1237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 06 21:08:47.179447 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:47.178983 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-77zzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 06 21:08:47.179447 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:47.179085 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" podUID="90be9695-5aa7-4767-95a3-81acb10d1237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 06 21:08:48.183029 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:48.182971 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:08:48.184154 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:48.184133 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-77zzd" Jul 06 21:08:49.016882 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:49.016848 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-sk8mt" Jul 06 21:08:49.017110 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:08:49.016933 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-nrqcv" Jul 06 21:09:04.050339 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:04.050251 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-tzkfx"] Jul 06 21:09:04.054166 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:04.054143 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tzkfx" Jul 06 21:09:04.056997 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:04.056967 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 21:09:04.058040 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:04.057984 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 21:09:04.058040 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:04.057999 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6lnnj\"" Jul 06 21:09:04.058202 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:04.058022 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 06 21:09:04.066840 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:04.066809 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-tzkfx"] Jul 06 21:09:04.161796 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:04.161762 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a684730a-371b-44c5-8798-e6db60d97413-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tzkfx\" (UID: \"a684730a-371b-44c5-8798-e6db60d97413\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tzkfx" Jul 06 21:09:04.161796 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:04.161797 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a684730a-371b-44c5-8798-e6db60d97413-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tzkfx\" (UID: \"a684730a-371b-44c5-8798-e6db60d97413\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tzkfx" Jul 06 21:09:04.162063 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:04.161838 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a684730a-371b-44c5-8798-e6db60d97413-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tzkfx\" (UID: \"a684730a-371b-44c5-8798-e6db60d97413\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tzkfx" Jul 06 21:09:04.162063 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:04.161928 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdz72\" (UniqueName: \"kubernetes.io/projected/a684730a-371b-44c5-8798-e6db60d97413-kube-api-access-fdz72\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tzkfx\" (UID: \"a684730a-371b-44c5-8798-e6db60d97413\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tzkfx" Jul 06 21:09:04.162063 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:04.161966 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a684730a-371b-44c5-8798-e6db60d97413-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tzkfx\" (UID: \"a684730a-371b-44c5-8798-e6db60d97413\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tzkfx" Jul 06 21:09:04.162063 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:04.162002 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a684730a-371b-44c5-8798-e6db60d97413-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tzkfx\" (UID: \"a684730a-371b-44c5-8798-e6db60d97413\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tzkfx" Jul 06 21:09:04.263247 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:04.263200 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a684730a-371b-44c5-8798-e6db60d97413-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tzkfx\" (UID: \"a684730a-371b-44c5-8798-e6db60d97413\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tzkfx" Jul 06 21:09:04.263247 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:04.263248 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a684730a-371b-44c5-8798-e6db60d97413-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tzkfx\" (UID: \"a684730a-371b-44c5-8798-e6db60d97413\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tzkfx" Jul 06 21:09:04.263519 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:04.263389 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a684730a-371b-44c5-8798-e6db60d97413-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tzkfx\" (UID: \"a684730a-371b-44c5-8798-e6db60d97413\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tzkfx" Jul 06 21:09:04.263519 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:04.263460 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdz72\" (UniqueName: \"kubernetes.io/projected/a684730a-371b-44c5-8798-e6db60d97413-kube-api-access-fdz72\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tzkfx\" (UID: \"a684730a-371b-44c5-8798-e6db60d97413\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tzkfx" Jul 06 21:09:04.263519 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:04.263499 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a684730a-371b-44c5-8798-e6db60d97413-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tzkfx\" (UID: \"a684730a-371b-44c5-8798-e6db60d97413\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tzkfx" Jul 06 21:09:04.263664 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:04.263543 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a684730a-371b-44c5-8798-e6db60d97413-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tzkfx\" (UID: \"a684730a-371b-44c5-8798-e6db60d97413\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tzkfx" Jul 06 21:09:04.263664 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:04.263634 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a684730a-371b-44c5-8798-e6db60d97413-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tzkfx\" (UID: \"a684730a-371b-44c5-8798-e6db60d97413\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tzkfx" Jul 06 21:09:04.263826 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:04.263692 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a684730a-371b-44c5-8798-e6db60d97413-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tzkfx\" (UID: \"a684730a-371b-44c5-8798-e6db60d97413\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tzkfx" Jul 06 21:09:04.263886 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:04.263826 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a684730a-371b-44c5-8798-e6db60d97413-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tzkfx\" (UID: \"a684730a-371b-44c5-8798-e6db60d97413\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tzkfx" Jul 06 21:09:04.265733 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:04.265700 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a684730a-371b-44c5-8798-e6db60d97413-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tzkfx\" (UID: \"a684730a-371b-44c5-8798-e6db60d97413\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tzkfx" Jul 06 21:09:04.266151 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:04.266118 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a684730a-371b-44c5-8798-e6db60d97413-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tzkfx\" (UID: \"a684730a-371b-44c5-8798-e6db60d97413\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tzkfx" Jul 06 21:09:04.274579 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:04.274551 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdz72\" (UniqueName: \"kubernetes.io/projected/a684730a-371b-44c5-8798-e6db60d97413-kube-api-access-fdz72\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tzkfx\" (UID: \"a684730a-371b-44c5-8798-e6db60d97413\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tzkfx" Jul 06 21:09:04.366276 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:04.366181 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tzkfx" Jul 06 21:09:04.514485 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:04.514452 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-tzkfx"] Jul 06 21:09:04.516138 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:09:04.516106 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda684730a_371b_44c5_8798_e6db60d97413.slice/crio-05bc92b419b5b5830cff31618fbb6eb608b533130034757796638ffc8caa4236 WatchSource:0}: Error finding container 05bc92b419b5b5830cff31618fbb6eb608b533130034757796638ffc8caa4236: Status 404 returned error can't find the container with id 05bc92b419b5b5830cff31618fbb6eb608b533130034757796638ffc8caa4236 Jul 06 21:09:04.517466 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:04.517444 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 21:09:05.167605 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:05.167528 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tzkfx" event={"ID":"a684730a-371b-44c5-8798-e6db60d97413","Type":"ContainerStarted","Data":"05bc92b419b5b5830cff31618fbb6eb608b533130034757796638ffc8caa4236"} Jul 06 21:09:06.174119 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:06.174080 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tzkfx" event={"ID":"a684730a-371b-44c5-8798-e6db60d97413","Type":"ContainerStarted","Data":"1efd280b9eb72a844baaadf837fb2178e117f20a9400617a680c8a235fc8afbc"} Jul 06 21:09:06.198113 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:06.198042 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tzkfx" podStartSLOduration=1.049177339 podStartE2EDuration="2.197992615s" totalImagesPullingTime="1.148815276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:09:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:09:06.196079103 +0000 UTC m=+630.514302364" watchObservedRunningTime="2026-07-06 21:09:06.197992615 +0000 UTC m=+630.516215876" Jul 06 21:09:14.366596 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:14.366554 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tzkfx" Jul 06 21:09:14.367107 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:14.366628 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tzkfx" Jul 06 21:09:14.379659 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:14.379629 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tzkfx" Jul 06 21:09:15.230036 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:15.229988 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tzkfx" Jul 06 21:09:24.353635 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:24.353589 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-8wbq9"] Jul 06 21:09:24.359323 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:24.359300 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8wbq9" Jul 06 21:09:24.362279 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:24.362254 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 06 21:09:24.368099 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:24.368074 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-8wbq9"] Jul 06 21:09:24.456942 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:24.456902 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4mgq\" (UniqueName: \"kubernetes.io/projected/976f5cf5-f24c-4578-aa50-beeafa06330f-kube-api-access-p4mgq\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8wbq9\" (UID: \"976f5cf5-f24c-4578-aa50-beeafa06330f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8wbq9" Jul 06 21:09:24.457149 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:24.456950 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/976f5cf5-f24c-4578-aa50-beeafa06330f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8wbq9\" (UID: \"976f5cf5-f24c-4578-aa50-beeafa06330f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8wbq9" Jul 06 21:09:24.457149 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:24.456976 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/976f5cf5-f24c-4578-aa50-beeafa06330f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8wbq9\" (UID: \"976f5cf5-f24c-4578-aa50-beeafa06330f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8wbq9" Jul 06 21:09:24.457149 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:24.457130 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/976f5cf5-f24c-4578-aa50-beeafa06330f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8wbq9\" (UID: \"976f5cf5-f24c-4578-aa50-beeafa06330f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8wbq9" Jul 06 21:09:24.457265 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:24.457165 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/976f5cf5-f24c-4578-aa50-beeafa06330f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8wbq9\" (UID: \"976f5cf5-f24c-4578-aa50-beeafa06330f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8wbq9" Jul 06 21:09:24.457265 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:24.457220 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/976f5cf5-f24c-4578-aa50-beeafa06330f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8wbq9\" (UID: \"976f5cf5-f24c-4578-aa50-beeafa06330f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8wbq9" Jul 06 21:09:24.558455 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:24.558408 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/976f5cf5-f24c-4578-aa50-beeafa06330f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8wbq9\" (UID: \"976f5cf5-f24c-4578-aa50-beeafa06330f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8wbq9" Jul 06 21:09:24.558455 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:24.558458 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4mgq\" (UniqueName: \"kubernetes.io/projected/976f5cf5-f24c-4578-aa50-beeafa06330f-kube-api-access-p4mgq\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8wbq9\" (UID: \"976f5cf5-f24c-4578-aa50-beeafa06330f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8wbq9" Jul 06 21:09:24.558723 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:24.558580 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/976f5cf5-f24c-4578-aa50-beeafa06330f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8wbq9\" (UID: \"976f5cf5-f24c-4578-aa50-beeafa06330f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8wbq9" Jul 06 21:09:24.558723 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:24.558625 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/976f5cf5-f24c-4578-aa50-beeafa06330f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8wbq9\" (UID: \"976f5cf5-f24c-4578-aa50-beeafa06330f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8wbq9" Jul 06 21:09:24.558829 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:24.558759 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/976f5cf5-f24c-4578-aa50-beeafa06330f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8wbq9\" (UID: \"976f5cf5-f24c-4578-aa50-beeafa06330f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8wbq9" Jul 06 21:09:24.558829 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:24.558788 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/976f5cf5-f24c-4578-aa50-beeafa06330f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8wbq9\" (UID: \"976f5cf5-f24c-4578-aa50-beeafa06330f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8wbq9" Jul 06 21:09:24.558933 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:24.558900 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/976f5cf5-f24c-4578-aa50-beeafa06330f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8wbq9\" (UID: \"976f5cf5-f24c-4578-aa50-beeafa06330f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8wbq9" Jul 06 21:09:24.559055 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:24.559035 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/976f5cf5-f24c-4578-aa50-beeafa06330f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8wbq9\" (UID: \"976f5cf5-f24c-4578-aa50-beeafa06330f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8wbq9" Jul 06 21:09:24.559101 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:24.559079 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/976f5cf5-f24c-4578-aa50-beeafa06330f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8wbq9\" (UID: \"976f5cf5-f24c-4578-aa50-beeafa06330f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8wbq9" Jul 06 21:09:24.561091 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:24.561068 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/976f5cf5-f24c-4578-aa50-beeafa06330f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8wbq9\" (UID: \"976f5cf5-f24c-4578-aa50-beeafa06330f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8wbq9" Jul 06 21:09:24.561242 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:24.561223 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/976f5cf5-f24c-4578-aa50-beeafa06330f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8wbq9\" (UID: \"976f5cf5-f24c-4578-aa50-beeafa06330f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8wbq9" Jul 06 21:09:24.575969 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:24.575938 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4mgq\" (UniqueName: \"kubernetes.io/projected/976f5cf5-f24c-4578-aa50-beeafa06330f-kube-api-access-p4mgq\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8wbq9\" (UID: \"976f5cf5-f24c-4578-aa50-beeafa06330f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8wbq9" Jul 06 21:09:24.671827 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:24.671785 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8wbq9" Jul 06 21:09:24.821957 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:24.821900 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-8wbq9"] Jul 06 21:09:24.824394 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:09:24.824362 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod976f5cf5_f24c_4578_aa50_beeafa06330f.slice/crio-8a91e2b97388b5db308619d1c13417f9b9e15d00ab8969146c9d9e7b1692dc33 WatchSource:0}: Error finding container 8a91e2b97388b5db308619d1c13417f9b9e15d00ab8969146c9d9e7b1692dc33: Status 404 returned error can't find the container with id 8a91e2b97388b5db308619d1c13417f9b9e15d00ab8969146c9d9e7b1692dc33 Jul 06 21:09:25.269276 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:25.269177 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8wbq9" event={"ID":"976f5cf5-f24c-4578-aa50-beeafa06330f","Type":"ContainerStarted","Data":"09a38e44027b834b0644968373363e87fff700587f2cd0c660939f4e6848a9c5"} Jul 06 21:09:25.269276 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:25.269232 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8wbq9" event={"ID":"976f5cf5-f24c-4578-aa50-beeafa06330f","Type":"ContainerStarted","Data":"8a91e2b97388b5db308619d1c13417f9b9e15d00ab8969146c9d9e7b1692dc33"} Jul 06 21:09:25.290049 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:25.289967 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8wbq9" podStartSLOduration=1.108514372 podStartE2EDuration="1.289950147s" totalImagesPullingTime="181.435775ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:09:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:09:25.288459706 +0000 UTC m=+649.606682968" watchObservedRunningTime="2026-07-06 21:09:25.289950147 +0000 UTC m=+649.608173452" Jul 06 21:09:34.672784 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:34.672736 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8wbq9" Jul 06 21:09:34.673314 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:34.672804 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8wbq9" Jul 06 21:09:34.685542 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:34.685506 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8wbq9" Jul 06 21:09:35.330713 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:09:35.330683 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8wbq9" Jul 06 21:11:17.502405 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:11:17.502359 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b8ddb49c7-t2682"] Jul 06 21:11:17.503805 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:11:17.503150 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5b8ddb49c7-t2682" podUID="211614d0-f797-46df-9888-2cd5b66e0c45" containerName="authorino" containerID="cri-o://15681dca261b8136dc9d416a849ac4e9062771123ff29433fffedc22f47f8038" gracePeriod=30 Jul 06 21:11:17.760419 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:11:17.760336 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b8ddb49c7-t2682" Jul 06 21:11:17.829663 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:11:17.829622 2577 generic.go:356] "Generic (PLEG): container finished" podID="211614d0-f797-46df-9888-2cd5b66e0c45" containerID="15681dca261b8136dc9d416a849ac4e9062771123ff29433fffedc22f47f8038" exitCode=0 Jul 06 21:11:17.829842 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:11:17.829672 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b8ddb49c7-t2682" Jul 06 21:11:17.829842 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:11:17.829670 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b8ddb49c7-t2682" event={"ID":"211614d0-f797-46df-9888-2cd5b66e0c45","Type":"ContainerDied","Data":"15681dca261b8136dc9d416a849ac4e9062771123ff29433fffedc22f47f8038"} Jul 06 21:11:17.829842 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:11:17.829714 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b8ddb49c7-t2682" event={"ID":"211614d0-f797-46df-9888-2cd5b66e0c45","Type":"ContainerDied","Data":"ce31dd8eca1b073d49ba8f028a14c2bc5a215bfafe19c5213297390e1345cb88"} Jul 06 21:11:17.829842 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:11:17.829737 2577 scope.go:117] "RemoveContainer" containerID="15681dca261b8136dc9d416a849ac4e9062771123ff29433fffedc22f47f8038" Jul 06 21:11:17.838737 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:11:17.838716 2577 scope.go:117] "RemoveContainer" containerID="15681dca261b8136dc9d416a849ac4e9062771123ff29433fffedc22f47f8038" Jul 06 21:11:17.839109 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:11:17.839088 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15681dca261b8136dc9d416a849ac4e9062771123ff29433fffedc22f47f8038\": container with ID starting with 15681dca261b8136dc9d416a849ac4e9062771123ff29433fffedc22f47f8038 not found: ID does not exist" containerID="15681dca261b8136dc9d416a849ac4e9062771123ff29433fffedc22f47f8038" Jul 06 21:11:17.839160 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:11:17.839120 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15681dca261b8136dc9d416a849ac4e9062771123ff29433fffedc22f47f8038"} err="failed to get container status \"15681dca261b8136dc9d416a849ac4e9062771123ff29433fffedc22f47f8038\": rpc error: code = NotFound desc = could not find container \"15681dca261b8136dc9d416a849ac4e9062771123ff29433fffedc22f47f8038\": container with ID starting with 15681dca261b8136dc9d416a849ac4e9062771123ff29433fffedc22f47f8038 not found: ID does not exist" Jul 06 21:11:17.936659 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:11:17.936626 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/211614d0-f797-46df-9888-2cd5b66e0c45-tls-cert\") pod \"211614d0-f797-46df-9888-2cd5b66e0c45\" (UID: \"211614d0-f797-46df-9888-2cd5b66e0c45\") " Jul 06 21:11:17.936853 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:11:17.936738 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bt58v\" (UniqueName: \"kubernetes.io/projected/211614d0-f797-46df-9888-2cd5b66e0c45-kube-api-access-bt58v\") pod \"211614d0-f797-46df-9888-2cd5b66e0c45\" (UID: \"211614d0-f797-46df-9888-2cd5b66e0c45\") " Jul 06 21:11:17.938856 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:11:17.938813 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/211614d0-f797-46df-9888-2cd5b66e0c45-kube-api-access-bt58v" (OuterVolumeSpecName: "kube-api-access-bt58v") pod "211614d0-f797-46df-9888-2cd5b66e0c45" (UID: "211614d0-f797-46df-9888-2cd5b66e0c45"). InnerVolumeSpecName "kube-api-access-bt58v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:11:17.948661 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:11:17.948619 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/211614d0-f797-46df-9888-2cd5b66e0c45-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "211614d0-f797-46df-9888-2cd5b66e0c45" (UID: "211614d0-f797-46df-9888-2cd5b66e0c45"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:11:18.038492 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:11:18.038393 2577 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/211614d0-f797-46df-9888-2cd5b66e0c45-tls-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:11:18.038492 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:11:18.038432 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt58v\" (UniqueName: \"kubernetes.io/projected/211614d0-f797-46df-9888-2cd5b66e0c45-kube-api-access-bt58v\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:11:18.157098 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:11:18.157049 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b8ddb49c7-t2682"] Jul 06 21:11:18.168910 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:11:18.168875 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5b8ddb49c7-t2682"] Jul 06 21:11:20.169401 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:11:20.169364 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="211614d0-f797-46df-9888-2cd5b66e0c45" path="/var/lib/kubelet/pods/211614d0-f797-46df-9888-2cd5b66e0c45/volumes" Jul 06 21:12:28.773406 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:28.773365 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67f746bd7b-b2kkn"] Jul 06 21:12:28.773981 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:28.773642 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-67f746bd7b-b2kkn" podUID="2e7b76dc-5003-4a19-a18d-7d3c6c375be1" containerName="manager" containerID="cri-o://6a839025ec504bc0e3f7bf50c55fd3d306f54d0801c24d8c339e0e6c63bdf675" gracePeriod=30 Jul 06 21:12:29.028351 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:29.028277 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67f746bd7b-b2kkn" Jul 06 21:12:29.089097 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:29.089064 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e7b76dc-5003-4a19-a18d-7d3c6c375be1-webhook-cert\") pod \"2e7b76dc-5003-4a19-a18d-7d3c6c375be1\" (UID: \"2e7b76dc-5003-4a19-a18d-7d3c6c375be1\") " Jul 06 21:12:29.089270 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:29.089166 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kz8f\" (UniqueName: \"kubernetes.io/projected/2e7b76dc-5003-4a19-a18d-7d3c6c375be1-kube-api-access-9kz8f\") pod \"2e7b76dc-5003-4a19-a18d-7d3c6c375be1\" (UID: \"2e7b76dc-5003-4a19-a18d-7d3c6c375be1\") " Jul 06 21:12:29.091272 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:29.091234 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e7b76dc-5003-4a19-a18d-7d3c6c375be1-kube-api-access-9kz8f" (OuterVolumeSpecName: "kube-api-access-9kz8f") pod "2e7b76dc-5003-4a19-a18d-7d3c6c375be1" (UID: "2e7b76dc-5003-4a19-a18d-7d3c6c375be1"). InnerVolumeSpecName "kube-api-access-9kz8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:12:29.091272 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:29.091243 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e7b76dc-5003-4a19-a18d-7d3c6c375be1-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2e7b76dc-5003-4a19-a18d-7d3c6c375be1" (UID: "2e7b76dc-5003-4a19-a18d-7d3c6c375be1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:12:29.174374 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:29.174337 2577 generic.go:356] "Generic (PLEG): container finished" podID="2e7b76dc-5003-4a19-a18d-7d3c6c375be1" containerID="6a839025ec504bc0e3f7bf50c55fd3d306f54d0801c24d8c339e0e6c63bdf675" exitCode=0 Jul 06 21:12:29.174559 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:29.174379 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67f746bd7b-b2kkn" Jul 06 21:12:29.174559 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:29.174411 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67f746bd7b-b2kkn" event={"ID":"2e7b76dc-5003-4a19-a18d-7d3c6c375be1","Type":"ContainerDied","Data":"6a839025ec504bc0e3f7bf50c55fd3d306f54d0801c24d8c339e0e6c63bdf675"} Jul 06 21:12:29.174559 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:29.174479 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67f746bd7b-b2kkn" event={"ID":"2e7b76dc-5003-4a19-a18d-7d3c6c375be1","Type":"ContainerDied","Data":"dfe76a703b13d0c21fe0f55b3df638f5323a794ff30fd1dcdba421d21248aadb"} Jul 06 21:12:29.174559 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:29.174508 2577 scope.go:117] "RemoveContainer" containerID="6a839025ec504bc0e3f7bf50c55fd3d306f54d0801c24d8c339e0e6c63bdf675" Jul 06 21:12:29.184640 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:29.184614 2577 scope.go:117] "RemoveContainer" containerID="6a839025ec504bc0e3f7bf50c55fd3d306f54d0801c24d8c339e0e6c63bdf675" Jul 06 21:12:29.185043 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:12:29.184997 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a839025ec504bc0e3f7bf50c55fd3d306f54d0801c24d8c339e0e6c63bdf675\": container with ID starting with 6a839025ec504bc0e3f7bf50c55fd3d306f54d0801c24d8c339e0e6c63bdf675 not found: ID does not exist" containerID="6a839025ec504bc0e3f7bf50c55fd3d306f54d0801c24d8c339e0e6c63bdf675" Jul 06 21:12:29.185105 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:29.185058 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a839025ec504bc0e3f7bf50c55fd3d306f54d0801c24d8c339e0e6c63bdf675"} err="failed to get container status \"6a839025ec504bc0e3f7bf50c55fd3d306f54d0801c24d8c339e0e6c63bdf675\": rpc error: code = NotFound desc = could not find container \"6a839025ec504bc0e3f7bf50c55fd3d306f54d0801c24d8c339e0e6c63bdf675\": container with ID starting with 6a839025ec504bc0e3f7bf50c55fd3d306f54d0801c24d8c339e0e6c63bdf675 not found: ID does not exist" Jul 06 21:12:29.189924 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:29.189900 2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e7b76dc-5003-4a19-a18d-7d3c6c375be1-webhook-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:12:29.189924 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:29.189922 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kz8f\" (UniqueName: \"kubernetes.io/projected/2e7b76dc-5003-4a19-a18d-7d3c6c375be1-kube-api-access-9kz8f\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:12:29.203253 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:29.203222 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67f746bd7b-b2kkn"] Jul 06 21:12:29.208419 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:29.208390 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-67f746bd7b-b2kkn"] Jul 06 21:12:30.120190 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:30.120149 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-67f746bd7b-tgt54"] Jul 06 21:12:30.120593 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:30.120545 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e7b76dc-5003-4a19-a18d-7d3c6c375be1" containerName="manager" Jul 06 21:12:30.120593 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:30.120556 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e7b76dc-5003-4a19-a18d-7d3c6c375be1" containerName="manager" Jul 06 21:12:30.120593 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:30.120569 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="211614d0-f797-46df-9888-2cd5b66e0c45" containerName="authorino" Jul 06 21:12:30.120593 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:30.120574 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="211614d0-f797-46df-9888-2cd5b66e0c45" containerName="authorino" Jul 06 21:12:30.120723 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:30.120642 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="211614d0-f797-46df-9888-2cd5b66e0c45" containerName="authorino" Jul 06 21:12:30.120723 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:30.120650 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e7b76dc-5003-4a19-a18d-7d3c6c375be1" containerName="manager" Jul 06 21:12:30.125281 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:30.125256 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67f746bd7b-tgt54" Jul 06 21:12:30.128843 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:30.128821 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 21:12:30.128982 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:30.128821 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 21:12:30.128982 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:30.128821 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5cnb8\"" Jul 06 21:12:30.134694 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:30.134661 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67f746bd7b-tgt54"] Jul 06 21:12:30.168786 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:30.168751 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e7b76dc-5003-4a19-a18d-7d3c6c375be1" path="/var/lib/kubelet/pods/2e7b76dc-5003-4a19-a18d-7d3c6c375be1/volumes" Jul 06 21:12:30.198812 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:30.198766 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/406c8c90-2de7-4516-b7e0-22a23bc828f2-webhook-cert\") pod \"maas-controller-67f746bd7b-tgt54\" (UID: \"406c8c90-2de7-4516-b7e0-22a23bc828f2\") " pod="opendatahub/maas-controller-67f746bd7b-tgt54" Jul 06 21:12:30.198996 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:30.198831 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjcds\" (UniqueName: \"kubernetes.io/projected/406c8c90-2de7-4516-b7e0-22a23bc828f2-kube-api-access-mjcds\") pod \"maas-controller-67f746bd7b-tgt54\" (UID: \"406c8c90-2de7-4516-b7e0-22a23bc828f2\") " pod="opendatahub/maas-controller-67f746bd7b-tgt54" Jul 06 21:12:30.300196 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:30.300141 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/406c8c90-2de7-4516-b7e0-22a23bc828f2-webhook-cert\") pod \"maas-controller-67f746bd7b-tgt54\" (UID: \"406c8c90-2de7-4516-b7e0-22a23bc828f2\") " pod="opendatahub/maas-controller-67f746bd7b-tgt54" Jul 06 21:12:30.300386 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:30.300216 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjcds\" (UniqueName: \"kubernetes.io/projected/406c8c90-2de7-4516-b7e0-22a23bc828f2-kube-api-access-mjcds\") pod \"maas-controller-67f746bd7b-tgt54\" (UID: \"406c8c90-2de7-4516-b7e0-22a23bc828f2\") " pod="opendatahub/maas-controller-67f746bd7b-tgt54" Jul 06 21:12:30.302863 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:30.302824 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/406c8c90-2de7-4516-b7e0-22a23bc828f2-webhook-cert\") pod \"maas-controller-67f746bd7b-tgt54\" (UID: \"406c8c90-2de7-4516-b7e0-22a23bc828f2\") " pod="opendatahub/maas-controller-67f746bd7b-tgt54" Jul 06 21:12:30.310096 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:30.310056 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjcds\" (UniqueName: \"kubernetes.io/projected/406c8c90-2de7-4516-b7e0-22a23bc828f2-kube-api-access-mjcds\") pod \"maas-controller-67f746bd7b-tgt54\" (UID: \"406c8c90-2de7-4516-b7e0-22a23bc828f2\") " pod="opendatahub/maas-controller-67f746bd7b-tgt54" Jul 06 21:12:30.437093 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:30.437052 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67f746bd7b-tgt54" Jul 06 21:12:30.578577 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:30.578550 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67f746bd7b-tgt54"] Jul 06 21:12:30.580848 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:12:30.580820 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod406c8c90_2de7_4516_b7e0_22a23bc828f2.slice/crio-ef325d0756333d6868d7c6a7880f39bc2b29a12fdcbc816872f86075ccb2110a WatchSource:0}: Error finding container ef325d0756333d6868d7c6a7880f39bc2b29a12fdcbc816872f86075ccb2110a: Status 404 returned error can't find the container with id ef325d0756333d6868d7c6a7880f39bc2b29a12fdcbc816872f86075ccb2110a Jul 06 21:12:31.186172 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:31.186135 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67f746bd7b-tgt54" event={"ID":"406c8c90-2de7-4516-b7e0-22a23bc828f2","Type":"ContainerStarted","Data":"c05dc5b911c9d7b686be7182cb180466ac63cc171b43f635fb5cfd4e6b2d3ba3"} Jul 06 21:12:31.186601 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:31.186335 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-67f746bd7b-tgt54" Jul 06 21:12:31.186601 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:31.186382 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67f746bd7b-tgt54" event={"ID":"406c8c90-2de7-4516-b7e0-22a23bc828f2","Type":"ContainerStarted","Data":"ef325d0756333d6868d7c6a7880f39bc2b29a12fdcbc816872f86075ccb2110a"} Jul 06 21:12:31.205753 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:31.205687 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-67f746bd7b-tgt54" podStartSLOduration=0.701326258 podStartE2EDuration="1.205667642s" totalImagesPullingTime="504.341384ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:12:31.203336038 +0000 UTC m=+835.521559289" watchObservedRunningTime="2026-07-06 21:12:31.205667642 +0000 UTC m=+835.523890902" Jul 06 21:12:42.198747 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:12:42.198707 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-67f746bd7b-tgt54" Jul 06 21:13:36.190529 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:13:36.190498 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qjs2n_64b45669-ed56-41b9-a3ed-1fe5c13508b6/ovn-acl-logging/0.log" Jul 06 21:13:36.192112 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:13:36.192092 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qjs2n_64b45669-ed56-41b9-a3ed-1fe5c13508b6/ovn-acl-logging/0.log" Jul 06 21:15:00.151080 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:15:00.150982 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722875-nmtng"] Jul 06 21:15:00.154329 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:15:00.154309 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722875-nmtng" Jul 06 21:15:00.181872 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:15:00.181838 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722875-nmtng"] Jul 06 21:15:00.250287 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:15:00.250247 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6dk8\" (UniqueName: \"kubernetes.io/projected/ce7f9d1b-993b-44dc-9972-e708a95283d9-kube-api-access-c6dk8\") pod \"maas-api-key-cleanup-29722875-nmtng\" (UID: \"ce7f9d1b-993b-44dc-9972-e708a95283d9\") " pod="opendatahub/maas-api-key-cleanup-29722875-nmtng" Jul 06 21:15:00.351621 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:15:00.351571 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6dk8\" (UniqueName: \"kubernetes.io/projected/ce7f9d1b-993b-44dc-9972-e708a95283d9-kube-api-access-c6dk8\") pod \"maas-api-key-cleanup-29722875-nmtng\" (UID: \"ce7f9d1b-993b-44dc-9972-e708a95283d9\") " pod="opendatahub/maas-api-key-cleanup-29722875-nmtng" Jul 06 21:15:00.361926 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:15:00.361900 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6dk8\" (UniqueName: \"kubernetes.io/projected/ce7f9d1b-993b-44dc-9972-e708a95283d9-kube-api-access-c6dk8\") pod \"maas-api-key-cleanup-29722875-nmtng\" (UID: \"ce7f9d1b-993b-44dc-9972-e708a95283d9\") " pod="opendatahub/maas-api-key-cleanup-29722875-nmtng" Jul 06 21:15:00.464493 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:15:00.464397 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722875-nmtng" Jul 06 21:15:00.608248 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:15:00.608221 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722875-nmtng"] Jul 06 21:15:00.610420 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:15:00.610390 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce7f9d1b_993b_44dc_9972_e708a95283d9.slice/crio-5de88015fe397af271680f071f4212cf6623b56b35496a83e2a9c8630a1245d6 WatchSource:0}: Error finding container 5de88015fe397af271680f071f4212cf6623b56b35496a83e2a9c8630a1245d6: Status 404 returned error can't find the container with id 5de88015fe397af271680f071f4212cf6623b56b35496a83e2a9c8630a1245d6 Jul 06 21:15:00.612403 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:15:00.612384 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 21:15:00.955436 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:15:00.955396 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722875-nmtng" event={"ID":"ce7f9d1b-993b-44dc-9972-e708a95283d9","Type":"ContainerStarted","Data":"5de88015fe397af271680f071f4212cf6623b56b35496a83e2a9c8630a1245d6"} Jul 06 21:15:02.966778 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:15:02.966738 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722875-nmtng" event={"ID":"ce7f9d1b-993b-44dc-9972-e708a95283d9","Type":"ContainerStarted","Data":"b6d34c6ba2f434aa4c26937cfc5e33df9ade68974db1799b250a1541243f9507"} Jul 06 21:15:02.989289 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:15:02.989232 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29722875-nmtng" podStartSLOduration=0.960430834 podStartE2EDuration="2.989215856s" totalImagesPullingTime="2.028785022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:15:02.988545836 +0000 UTC m=+987.306769115" watchObservedRunningTime="2026-07-06 21:15:02.989215856 +0000 UTC m=+987.307439149" Jul 06 21:15:03.973806 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:15:03.973713 2577 generic.go:356] "Generic (PLEG): container finished" podID="ce7f9d1b-993b-44dc-9972-e708a95283d9" containerID="b6d34c6ba2f434aa4c26937cfc5e33df9ade68974db1799b250a1541243f9507" exitCode=0 Jul 06 21:15:03.973806 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:15:03.973793 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722875-nmtng" event={"ID":"ce7f9d1b-993b-44dc-9972-e708a95283d9","Type":"ContainerDied","Data":"b6d34c6ba2f434aa4c26937cfc5e33df9ade68974db1799b250a1541243f9507"} Jul 06 21:15:05.121675 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:15:05.121646 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722875-nmtng" Jul 06 21:15:05.200094 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:15:05.200061 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6dk8\" (UniqueName: \"kubernetes.io/projected/ce7f9d1b-993b-44dc-9972-e708a95283d9-kube-api-access-c6dk8\") pod \"ce7f9d1b-993b-44dc-9972-e708a95283d9\" (UID: \"ce7f9d1b-993b-44dc-9972-e708a95283d9\") " Jul 06 21:15:05.202157 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:15:05.202132 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce7f9d1b-993b-44dc-9972-e708a95283d9-kube-api-access-c6dk8" (OuterVolumeSpecName: "kube-api-access-c6dk8") pod "ce7f9d1b-993b-44dc-9972-e708a95283d9" (UID: "ce7f9d1b-993b-44dc-9972-e708a95283d9"). InnerVolumeSpecName "kube-api-access-c6dk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:15:05.301746 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:15:05.301644 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6dk8\" (UniqueName: \"kubernetes.io/projected/ce7f9d1b-993b-44dc-9972-e708a95283d9-kube-api-access-c6dk8\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:15:05.985159 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:15:05.985126 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722875-nmtng" Jul 06 21:15:05.985338 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:15:05.985122 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722875-nmtng" event={"ID":"ce7f9d1b-993b-44dc-9972-e708a95283d9","Type":"ContainerDied","Data":"5de88015fe397af271680f071f4212cf6623b56b35496a83e2a9c8630a1245d6"} Jul 06 21:15:05.985338 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:15:05.985249 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5de88015fe397af271680f071f4212cf6623b56b35496a83e2a9c8630a1245d6" Jul 06 21:18:36.222738 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:18:36.222710 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qjs2n_64b45669-ed56-41b9-a3ed-1fe5c13508b6/ovn-acl-logging/0.log" Jul 06 21:18:36.223889 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:18:36.223869 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qjs2n_64b45669-ed56-41b9-a3ed-1fe5c13508b6/ovn-acl-logging/0.log" Jul 06 21:23:36.252523 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:23:36.252489 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qjs2n_64b45669-ed56-41b9-a3ed-1fe5c13508b6/ovn-acl-logging/0.log" Jul 06 21:23:36.263909 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:23:36.263884 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qjs2n_64b45669-ed56-41b9-a3ed-1fe5c13508b6/ovn-acl-logging/0.log" Jul 06 21:24:37.357839 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.357807 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n"] Jul 06 21:24:37.358263 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.358219 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce7f9d1b-993b-44dc-9972-e708a95283d9" containerName="cleanup" Jul 06 21:24:37.358263 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.358233 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce7f9d1b-993b-44dc-9972-e708a95283d9" containerName="cleanup" Jul 06 21:24:37.358347 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.358314 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce7f9d1b-993b-44dc-9972-e708a95283d9" containerName="cleanup" Jul 06 21:24:37.361405 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.361387 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:37.365096 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.365069 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-38e0067e-openshift-default-dockercfg-ng6gq\"" Jul 06 21:24:37.388321 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.388281 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n"] Jul 06 21:24:37.440206 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.440168 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-credential-socket\") pod \"e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:37.440206 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.440222 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-istio-data\") pod \"e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:37.440485 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.440244 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-istio-podinfo\") pod \"e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:37.440485 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.440301 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-istio-envoy\") pod \"e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:37.440485 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.440374 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-istio-token\") pod \"e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:37.440485 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.440418 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-workload-certs\") pod \"e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:37.440485 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.440445 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-istiod-ca-cert\") pod \"e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:37.440485 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.440469 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xbrk\" (UniqueName: \"kubernetes.io/projected/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-kube-api-access-6xbrk\") pod \"e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:37.440485 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.440486 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-workload-socket\") pod \"e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:37.541173 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.541135 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-workload-socket\") pod \"e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:37.541346 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.541243 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-credential-socket\") pod \"e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:37.541346 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.541290 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-istio-data\") pod \"e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:37.541346 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.541322 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-istio-podinfo\") pod \"e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:37.541470 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.541349 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-istio-envoy\") pod \"e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:37.541470 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.541379 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-istio-token\") pod \"e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:37.541470 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.541416 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-workload-certs\") pod \"e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:37.541470 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.541445 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-istiod-ca-cert\") pod \"e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:37.541656 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.541486 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xbrk\" (UniqueName: \"kubernetes.io/projected/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-kube-api-access-6xbrk\") pod \"e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:37.541656 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.541614 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-workload-socket\") pod \"e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:37.541749 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.541669 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-credential-socket\") pod \"e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:37.541749 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.541705 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-istio-data\") pod \"e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:37.541906 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.541884 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-workload-certs\") pod \"e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:37.542223 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.542208 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-istiod-ca-cert\") pod \"e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:37.543658 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.543636 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-istio-envoy\") pod \"e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:37.543774 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.543759 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-istio-podinfo\") pod \"e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:37.552602 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.552574 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-istio-token\") pod \"e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:37.553419 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.553390 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xbrk\" (UniqueName: \"kubernetes.io/projected/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-kube-api-access-6xbrk\") pod \"e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:37.672193 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.672160 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:37.825187 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.825158 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n"] Jul 06 21:24:37.827377 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:24:37.827346 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2bb4c7e_b540_4708_9ba8_d3d53b26053c.slice/crio-e7087471f79d2fab85e87b6c7163415bacb2c0cb3d47f7f973a0bfe61de6f3b5 WatchSource:0}: Error finding container e7087471f79d2fab85e87b6c7163415bacb2c0cb3d47f7f973a0bfe61de6f3b5: Status 404 returned error can't find the container with id e7087471f79d2fab85e87b6c7163415bacb2c0cb3d47f7f973a0bfe61de6f3b5 Jul 06 21:24:37.829552 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.829526 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:24:37.829619 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.829607 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:24:37.829655 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.829634 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:24:37.875512 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:37.875478 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" event={"ID":"e2bb4c7e-b540-4708-9ba8-d3d53b26053c","Type":"ContainerStarted","Data":"e7087471f79d2fab85e87b6c7163415bacb2c0cb3d47f7f973a0bfe61de6f3b5"} Jul 06 21:24:38.656131 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:38.656091 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7"] Jul 06 21:24:38.659703 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:38.659682 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7" Jul 06 21:24:38.662376 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:38.662349 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-38e0067e-serving-cert\"" Jul 06 21:24:38.673466 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:38.673434 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7"] Jul 06 21:24:38.755231 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:38.755192 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqv7n\" (UniqueName: \"kubernetes.io/projected/d4b50453-f04d-4f44-bdbf-91132ebfefcb-kube-api-access-qqv7n\") pod \"maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7\" (UID: \"d4b50453-f04d-4f44-bdbf-91132ebfefcb\") " pod="opendatahub/maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7" Jul 06 21:24:38.755231 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:38.755258 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4b50453-f04d-4f44-bdbf-91132ebfefcb-openshift-service-ca\") pod \"maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7\" (UID: \"d4b50453-f04d-4f44-bdbf-91132ebfefcb\") " pod="opendatahub/maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7" Jul 06 21:24:38.755504 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:38.755385 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4b50453-f04d-4f44-bdbf-91132ebfefcb-maas-api-tls\") pod \"maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7\" (UID: \"d4b50453-f04d-4f44-bdbf-91132ebfefcb\") " pod="opendatahub/maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7" Jul 06 21:24:38.857078 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:38.857032 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqv7n\" (UniqueName: \"kubernetes.io/projected/d4b50453-f04d-4f44-bdbf-91132ebfefcb-kube-api-access-qqv7n\") pod \"maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7\" (UID: \"d4b50453-f04d-4f44-bdbf-91132ebfefcb\") " pod="opendatahub/maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7" Jul 06 21:24:38.857284 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:38.857099 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4b50453-f04d-4f44-bdbf-91132ebfefcb-openshift-service-ca\") pod \"maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7\" (UID: \"d4b50453-f04d-4f44-bdbf-91132ebfefcb\") " pod="opendatahub/maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7" Jul 06 21:24:38.857284 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:38.857137 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4b50453-f04d-4f44-bdbf-91132ebfefcb-maas-api-tls\") pod \"maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7\" (UID: \"d4b50453-f04d-4f44-bdbf-91132ebfefcb\") " pod="opendatahub/maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7" Jul 06 21:24:38.857406 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:24:38.857293 2577 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-38e0067e-serving-cert: secret "maas-api-e2e-ait-38e0067e-serving-cert" not found Jul 06 21:24:38.857406 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:24:38.857373 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4b50453-f04d-4f44-bdbf-91132ebfefcb-maas-api-tls podName:d4b50453-f04d-4f44-bdbf-91132ebfefcb nodeName:}" failed. No retries permitted until 2026-07-06 21:24:39.357350187 +0000 UTC m=+1563.675573428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d4b50453-f04d-4f44-bdbf-91132ebfefcb-maas-api-tls") pod "maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7" (UID: "d4b50453-f04d-4f44-bdbf-91132ebfefcb") : secret "maas-api-e2e-ait-38e0067e-serving-cert" not found Jul 06 21:24:38.857813 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:38.857790 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4b50453-f04d-4f44-bdbf-91132ebfefcb-openshift-service-ca\") pod \"maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7\" (UID: \"d4b50453-f04d-4f44-bdbf-91132ebfefcb\") " pod="opendatahub/maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7" Jul 06 21:24:38.882548 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:38.882512 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" event={"ID":"e2bb4c7e-b540-4708-9ba8-d3d53b26053c","Type":"ContainerStarted","Data":"29f189acf0146b8b3a065112bb6374c7ec188ae05b6e8e12975757f8d5ba76ec"} Jul 06 21:24:38.954332 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:38.954247 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqv7n\" (UniqueName: \"kubernetes.io/projected/d4b50453-f04d-4f44-bdbf-91132ebfefcb-kube-api-access-qqv7n\") pod \"maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7\" (UID: \"d4b50453-f04d-4f44-bdbf-91132ebfefcb\") " pod="opendatahub/maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7" Jul 06 21:24:39.091293 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:39.091228 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" podStartSLOduration=2.091205215 podStartE2EDuration="2.091205215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:24:39.089828137 +0000 UTC m=+1563.408051396" watchObservedRunningTime="2026-07-06 21:24:39.091205215 +0000 UTC m=+1563.409428477" Jul 06 21:24:39.363099 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:39.362987 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4b50453-f04d-4f44-bdbf-91132ebfefcb-maas-api-tls\") pod \"maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7\" (UID: \"d4b50453-f04d-4f44-bdbf-91132ebfefcb\") " pod="opendatahub/maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7" Jul 06 21:24:39.365764 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:39.365732 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4b50453-f04d-4f44-bdbf-91132ebfefcb-maas-api-tls\") pod \"maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7\" (UID: \"d4b50453-f04d-4f44-bdbf-91132ebfefcb\") " pod="opendatahub/maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7" Jul 06 21:24:39.572773 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:39.572719 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7" Jul 06 21:24:39.673396 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:39.673333 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:39.679153 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:39.679117 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:39.730132 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:39.730100 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7"] Jul 06 21:24:39.733186 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:24:39.733141 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b50453_f04d_4f44_bdbf_91132ebfefcb.slice/crio-99e6704af2c703e773ad45c5a4fe9ac902bef75abcca5ff2084d07d77058a4b8 WatchSource:0}: Error finding container 99e6704af2c703e773ad45c5a4fe9ac902bef75abcca5ff2084d07d77058a4b8: Status 404 returned error can't find the container with id 99e6704af2c703e773ad45c5a4fe9ac902bef75abcca5ff2084d07d77058a4b8 Jul 06 21:24:39.734780 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:39.734749 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 21:24:39.888377 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:39.888284 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7" event={"ID":"d4b50453-f04d-4f44-bdbf-91132ebfefcb","Type":"ContainerStarted","Data":"99e6704af2c703e773ad45c5a4fe9ac902bef75abcca5ff2084d07d77058a4b8"} Jul 06 21:24:39.888529 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:39.888493 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:39.889410 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:39.889390 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:40.895346 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:40.895308 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7" event={"ID":"d4b50453-f04d-4f44-bdbf-91132ebfefcb","Type":"ContainerStarted","Data":"bb59509c0f04bc34761cc5e7291ba8f3e5642aa95cda39556e41f274c848195a"} Jul 06 21:24:40.918372 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:40.918317 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7" podStartSLOduration=1.869134107 podStartE2EDuration="2.918298127s" totalImagesPullingTime="1.04916402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:24:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:24:40.915793503 +0000 UTC m=+1565.234016763" watchObservedRunningTime="2026-07-06 21:24:40.918298127 +0000 UTC m=+1565.236521445" Jul 06 21:24:41.900736 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:41.900706 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7" Jul 06 21:24:43.894677 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:43.894635 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n"] Jul 06 21:24:43.895154 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:43.894930 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" podUID="e2bb4c7e-b540-4708-9ba8-d3d53b26053c" containerName="istio-proxy" containerID="cri-o://29f189acf0146b8b3a065112bb6374c7ec188ae05b6e8e12975757f8d5ba76ec" gracePeriod=30 Jul 06 21:24:44.216882 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:44.216803 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7"] Jul 06 21:24:44.217082 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:44.216994 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7" podUID="d4b50453-f04d-4f44-bdbf-91132ebfefcb" containerName="maas-api" containerID="cri-o://bb59509c0f04bc34761cc5e7291ba8f3e5642aa95cda39556e41f274c848195a" gracePeriod=30 Jul 06 21:24:44.479279 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:44.479202 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7" Jul 06 21:24:44.617413 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:44.617374 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqv7n\" (UniqueName: \"kubernetes.io/projected/d4b50453-f04d-4f44-bdbf-91132ebfefcb-kube-api-access-qqv7n\") pod \"d4b50453-f04d-4f44-bdbf-91132ebfefcb\" (UID: \"d4b50453-f04d-4f44-bdbf-91132ebfefcb\") " Jul 06 21:24:44.617599 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:44.617489 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4b50453-f04d-4f44-bdbf-91132ebfefcb-maas-api-tls\") pod \"d4b50453-f04d-4f44-bdbf-91132ebfefcb\" (UID: \"d4b50453-f04d-4f44-bdbf-91132ebfefcb\") " Jul 06 21:24:44.617599 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:44.617542 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4b50453-f04d-4f44-bdbf-91132ebfefcb-openshift-service-ca\") pod \"d4b50453-f04d-4f44-bdbf-91132ebfefcb\" (UID: \"d4b50453-f04d-4f44-bdbf-91132ebfefcb\") " Jul 06 21:24:44.617925 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:44.617897 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4b50453-f04d-4f44-bdbf-91132ebfefcb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d4b50453-f04d-4f44-bdbf-91132ebfefcb" (UID: "d4b50453-f04d-4f44-bdbf-91132ebfefcb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:24:44.619634 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:44.619607 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4b50453-f04d-4f44-bdbf-91132ebfefcb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d4b50453-f04d-4f44-bdbf-91132ebfefcb" (UID: "d4b50453-f04d-4f44-bdbf-91132ebfefcb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:24:44.619719 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:44.619604 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b50453-f04d-4f44-bdbf-91132ebfefcb-kube-api-access-qqv7n" (OuterVolumeSpecName: "kube-api-access-qqv7n") pod "d4b50453-f04d-4f44-bdbf-91132ebfefcb" (UID: "d4b50453-f04d-4f44-bdbf-91132ebfefcb"). InnerVolumeSpecName "kube-api-access-qqv7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:24:44.718420 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:44.718371 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4b50453-f04d-4f44-bdbf-91132ebfefcb-maas-api-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:24:44.718420 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:44.718417 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4b50453-f04d-4f44-bdbf-91132ebfefcb-openshift-service-ca\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:24:44.718629 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:44.718431 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqv7n\" (UniqueName: \"kubernetes.io/projected/d4b50453-f04d-4f44-bdbf-91132ebfefcb-kube-api-access-qqv7n\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:24:44.925357 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:44.925320 2577 generic.go:356] "Generic (PLEG): container finished" podID="d4b50453-f04d-4f44-bdbf-91132ebfefcb" containerID="bb59509c0f04bc34761cc5e7291ba8f3e5642aa95cda39556e41f274c848195a" exitCode=0 Jul 06 21:24:44.925783 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:44.925384 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7" Jul 06 21:24:44.925783 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:44.925399 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7" event={"ID":"d4b50453-f04d-4f44-bdbf-91132ebfefcb","Type":"ContainerDied","Data":"bb59509c0f04bc34761cc5e7291ba8f3e5642aa95cda39556e41f274c848195a"} Jul 06 21:24:44.925783 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:44.925453 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7" event={"ID":"d4b50453-f04d-4f44-bdbf-91132ebfefcb","Type":"ContainerDied","Data":"99e6704af2c703e773ad45c5a4fe9ac902bef75abcca5ff2084d07d77058a4b8"} Jul 06 21:24:44.925783 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:44.925475 2577 scope.go:117] "RemoveContainer" containerID="bb59509c0f04bc34761cc5e7291ba8f3e5642aa95cda39556e41f274c848195a" Jul 06 21:24:44.935688 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:44.935664 2577 scope.go:117] "RemoveContainer" containerID="bb59509c0f04bc34761cc5e7291ba8f3e5642aa95cda39556e41f274c848195a" Jul 06 21:24:44.935985 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:24:44.935966 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb59509c0f04bc34761cc5e7291ba8f3e5642aa95cda39556e41f274c848195a\": container with ID starting with bb59509c0f04bc34761cc5e7291ba8f3e5642aa95cda39556e41f274c848195a not found: ID does not exist" containerID="bb59509c0f04bc34761cc5e7291ba8f3e5642aa95cda39556e41f274c848195a" Jul 06 21:24:44.936071 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:44.935996 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb59509c0f04bc34761cc5e7291ba8f3e5642aa95cda39556e41f274c848195a"} err="failed to get container status \"bb59509c0f04bc34761cc5e7291ba8f3e5642aa95cda39556e41f274c848195a\": rpc error: code = NotFound desc = could not find container \"bb59509c0f04bc34761cc5e7291ba8f3e5642aa95cda39556e41f274c848195a\": container with ID starting with bb59509c0f04bc34761cc5e7291ba8f3e5642aa95cda39556e41f274c848195a not found: ID does not exist" Jul 06 21:24:44.950339 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:44.950294 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7"] Jul 06 21:24:44.954883 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:44.954849 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-38e0067e-6f4489c8-qtlg7"] Jul 06 21:24:46.168967 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:46.168933 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4b50453-f04d-4f44-bdbf-91132ebfefcb" path="/var/lib/kubelet/pods/d4b50453-f04d-4f44-bdbf-91132ebfefcb/volumes" Jul 06 21:24:49.151472 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.151435 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:49.263805 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.263702 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xbrk\" (UniqueName: \"kubernetes.io/projected/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-kube-api-access-6xbrk\") pod \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " Jul 06 21:24:49.263805 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.263756 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-istiod-ca-cert\") pod \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " Jul 06 21:24:49.263805 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.263789 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-workload-certs\") pod \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " Jul 06 21:24:49.264078 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.263811 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-credential-socket\") pod \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " Jul 06 21:24:49.264078 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.263834 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-istio-token\") pod \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " Jul 06 21:24:49.264078 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.263881 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-istio-podinfo\") pod \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " Jul 06 21:24:49.264078 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.263915 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-istio-data\") pod \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " Jul 06 21:24:49.264078 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.263965 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-workload-socket\") pod \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " Jul 06 21:24:49.264078 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.264037 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-istio-envoy\") pod \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\" (UID: \"e2bb4c7e-b540-4708-9ba8-d3d53b26053c\") " Jul 06 21:24:49.264370 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.264147 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e2bb4c7e-b540-4708-9ba8-d3d53b26053c" (UID: "e2bb4c7e-b540-4708-9ba8-d3d53b26053c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:24:49.264370 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.264335 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-istio-data" (OuterVolumeSpecName: "istio-data") pod "e2bb4c7e-b540-4708-9ba8-d3d53b26053c" (UID: "e2bb4c7e-b540-4708-9ba8-d3d53b26053c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:24:49.264370 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.264159 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e2bb4c7e-b540-4708-9ba8-d3d53b26053c" (UID: "e2bb4c7e-b540-4708-9ba8-d3d53b26053c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:24:49.264370 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.264357 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e2bb4c7e-b540-4708-9ba8-d3d53b26053c" (UID: "e2bb4c7e-b540-4708-9ba8-d3d53b26053c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:24:49.264564 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.264360 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e2bb4c7e-b540-4708-9ba8-d3d53b26053c" (UID: "e2bb4c7e-b540-4708-9ba8-d3d53b26053c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:24:49.264564 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.264411 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-credential-socket\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:24:49.266322 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.266288 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-istio-token" (OuterVolumeSpecName: "istio-token") pod "e2bb4c7e-b540-4708-9ba8-d3d53b26053c" (UID: "e2bb4c7e-b540-4708-9ba8-d3d53b26053c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:24:49.266458 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.266399 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-kube-api-access-6xbrk" (OuterVolumeSpecName: "kube-api-access-6xbrk") pod "e2bb4c7e-b540-4708-9ba8-d3d53b26053c" (UID: "e2bb4c7e-b540-4708-9ba8-d3d53b26053c"). InnerVolumeSpecName "kube-api-access-6xbrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:24:49.266458 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.266444 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e2bb4c7e-b540-4708-9ba8-d3d53b26053c" (UID: "e2bb4c7e-b540-4708-9ba8-d3d53b26053c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 21:24:49.266590 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.266575 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e2bb4c7e-b540-4708-9ba8-d3d53b26053c" (UID: "e2bb4c7e-b540-4708-9ba8-d3d53b26053c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:24:49.365836 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.365798 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-istio-token\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:24:49.365836 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.365830 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-istio-podinfo\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:24:49.365836 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.365839 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-istio-data\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:24:49.366115 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.365863 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-workload-socket\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:24:49.366115 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.365873 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-istio-envoy\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:24:49.366115 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.365883 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xbrk\" (UniqueName: \"kubernetes.io/projected/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-kube-api-access-6xbrk\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:24:49.366115 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.365893 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-istiod-ca-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:24:49.366115 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.365903 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2bb4c7e-b540-4708-9ba8-d3d53b26053c-workload-certs\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:24:49.952653 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.952621 2577 generic.go:356] "Generic (PLEG): container finished" podID="e2bb4c7e-b540-4708-9ba8-d3d53b26053c" containerID="29f189acf0146b8b3a065112bb6374c7ec188ae05b6e8e12975757f8d5ba76ec" exitCode=0 Jul 06 21:24:49.952842 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.952670 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" Jul 06 21:24:49.952842 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.952700 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" event={"ID":"e2bb4c7e-b540-4708-9ba8-d3d53b26053c","Type":"ContainerDied","Data":"29f189acf0146b8b3a065112bb6374c7ec188ae05b6e8e12975757f8d5ba76ec"} Jul 06 21:24:49.952842 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.952760 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n" event={"ID":"e2bb4c7e-b540-4708-9ba8-d3d53b26053c","Type":"ContainerDied","Data":"e7087471f79d2fab85e87b6c7163415bacb2c0cb3d47f7f973a0bfe61de6f3b5"} Jul 06 21:24:49.952842 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.952783 2577 scope.go:117] "RemoveContainer" containerID="29f189acf0146b8b3a065112bb6374c7ec188ae05b6e8e12975757f8d5ba76ec" Jul 06 21:24:49.962092 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.962077 2577 scope.go:117] "RemoveContainer" containerID="29f189acf0146b8b3a065112bb6374c7ec188ae05b6e8e12975757f8d5ba76ec" Jul 06 21:24:49.962413 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:24:49.962379 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29f189acf0146b8b3a065112bb6374c7ec188ae05b6e8e12975757f8d5ba76ec\": container with ID starting with 29f189acf0146b8b3a065112bb6374c7ec188ae05b6e8e12975757f8d5ba76ec not found: ID does not exist" containerID="29f189acf0146b8b3a065112bb6374c7ec188ae05b6e8e12975757f8d5ba76ec" Jul 06 21:24:49.962466 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.962425 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29f189acf0146b8b3a065112bb6374c7ec188ae05b6e8e12975757f8d5ba76ec"} err="failed to get container status \"29f189acf0146b8b3a065112bb6374c7ec188ae05b6e8e12975757f8d5ba76ec\": rpc error: code = NotFound desc = could not find container \"29f189acf0146b8b3a065112bb6374c7ec188ae05b6e8e12975757f8d5ba76ec\": container with ID starting with 29f189acf0146b8b3a065112bb6374c7ec188ae05b6e8e12975757f8d5ba76ec not found: ID does not exist" Jul 06 21:24:49.976807 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.976772 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n"] Jul 06 21:24:49.979603 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:49.979575 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-38e0067e-openshift-default-8cdb47ff5-hxx5n"] Jul 06 21:24:50.169266 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.169231 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2bb4c7e-b540-4708-9ba8-d3d53b26053c" path="/var/lib/kubelet/pods/e2bb4c7e-b540-4708-9ba8-d3d53b26053c/volumes" Jul 06 21:24:50.664571 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.664531 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz"] Jul 06 21:24:50.665174 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.665156 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2bb4c7e-b540-4708-9ba8-d3d53b26053c" containerName="istio-proxy" Jul 06 21:24:50.665220 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.665178 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2bb4c7e-b540-4708-9ba8-d3d53b26053c" containerName="istio-proxy" Jul 06 21:24:50.665220 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.665198 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4b50453-f04d-4f44-bdbf-91132ebfefcb" containerName="maas-api" Jul 06 21:24:50.665220 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.665208 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4b50453-f04d-4f44-bdbf-91132ebfefcb" containerName="maas-api" Jul 06 21:24:50.665314 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.665308 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2bb4c7e-b540-4708-9ba8-d3d53b26053c" containerName="istio-proxy" Jul 06 21:24:50.665347 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.665323 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4b50453-f04d-4f44-bdbf-91132ebfefcb" containerName="maas-api" Jul 06 21:24:50.670293 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.670269 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:50.673240 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.673214 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-577d057f-openshift-default-dockercfg-zqsbv\"" Jul 06 21:24:50.681926 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.681895 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz"] Jul 06 21:24:50.778644 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.778610 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aabdb90e-9aff-4219-8270-de72d786149b-workload-socket\") pod \"e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:50.778644 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.778647 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aabdb90e-9aff-4219-8270-de72d786149b-istio-envoy\") pod \"e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:50.778881 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.778693 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aabdb90e-9aff-4219-8270-de72d786149b-workload-certs\") pod \"e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:50.778881 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.778712 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqctr\" (UniqueName: \"kubernetes.io/projected/aabdb90e-9aff-4219-8270-de72d786149b-kube-api-access-cqctr\") pod \"e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:50.778881 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.778764 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aabdb90e-9aff-4219-8270-de72d786149b-istio-token\") pod \"e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:50.778881 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.778785 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aabdb90e-9aff-4219-8270-de72d786149b-istio-data\") pod \"e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:50.778881 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.778811 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aabdb90e-9aff-4219-8270-de72d786149b-istio-podinfo\") pod \"e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:50.778881 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.778831 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aabdb90e-9aff-4219-8270-de72d786149b-credential-socket\") pod \"e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:50.779101 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.778940 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aabdb90e-9aff-4219-8270-de72d786149b-istiod-ca-cert\") pod \"e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:50.879836 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.879789 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aabdb90e-9aff-4219-8270-de72d786149b-istio-token\") pod \"e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:50.879836 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.879834 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aabdb90e-9aff-4219-8270-de72d786149b-istio-data\") pod \"e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:50.880117 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.879860 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aabdb90e-9aff-4219-8270-de72d786149b-istio-podinfo\") pod \"e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:50.880117 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.879912 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aabdb90e-9aff-4219-8270-de72d786149b-credential-socket\") pod \"e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:50.880117 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.879950 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aabdb90e-9aff-4219-8270-de72d786149b-istiod-ca-cert\") pod \"e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:50.880117 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.880038 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aabdb90e-9aff-4219-8270-de72d786149b-workload-socket\") pod \"e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:50.880117 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.880063 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aabdb90e-9aff-4219-8270-de72d786149b-istio-envoy\") pod \"e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:50.880347 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.880118 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aabdb90e-9aff-4219-8270-de72d786149b-workload-certs\") pod \"e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:50.880347 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.880140 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqctr\" (UniqueName: \"kubernetes.io/projected/aabdb90e-9aff-4219-8270-de72d786149b-kube-api-access-cqctr\") pod \"e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:50.880347 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.880306 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aabdb90e-9aff-4219-8270-de72d786149b-istio-data\") pod \"e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:50.880508 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.880481 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aabdb90e-9aff-4219-8270-de72d786149b-credential-socket\") pod \"e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:50.880586 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.880566 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aabdb90e-9aff-4219-8270-de72d786149b-workload-socket\") pod \"e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:50.880699 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.880668 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aabdb90e-9aff-4219-8270-de72d786149b-workload-certs\") pod \"e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:50.880836 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.880820 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aabdb90e-9aff-4219-8270-de72d786149b-istiod-ca-cert\") pod \"e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:50.882394 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.882365 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aabdb90e-9aff-4219-8270-de72d786149b-istio-envoy\") pod \"e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:50.882532 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.882513 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aabdb90e-9aff-4219-8270-de72d786149b-istio-podinfo\") pod \"e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:50.890755 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.890719 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aabdb90e-9aff-4219-8270-de72d786149b-istio-token\") pod \"e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:50.890938 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.890763 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqctr\" (UniqueName: \"kubernetes.io/projected/aabdb90e-9aff-4219-8270-de72d786149b-kube-api-access-cqctr\") pod \"e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:50.982466 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:50.982367 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:51.139292 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:51.139255 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz"] Jul 06 21:24:51.140907 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:24:51.140882 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaabdb90e_9aff_4219_8270_de72d786149b.slice/crio-0ed94421d43c9f9348396f6c58afaa415a5371d49faf2b03d69e562f8f332a38 WatchSource:0}: Error finding container 0ed94421d43c9f9348396f6c58afaa415a5371d49faf2b03d69e562f8f332a38: Status 404 returned error can't find the container with id 0ed94421d43c9f9348396f6c58afaa415a5371d49faf2b03d69e562f8f332a38 Jul 06 21:24:51.143752 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:51.143712 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:24:51.143830 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:51.143815 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:24:51.143875 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:51.143863 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:24:51.969025 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:51.968965 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" event={"ID":"aabdb90e-9aff-4219-8270-de72d786149b","Type":"ContainerStarted","Data":"41c40ad5943d62a642349c2b93a2cea573a749d233f626496bf9415c0cb056a3"} Jul 06 21:24:51.969519 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:51.969054 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" event={"ID":"aabdb90e-9aff-4219-8270-de72d786149b","Type":"ContainerStarted","Data":"0ed94421d43c9f9348396f6c58afaa415a5371d49faf2b03d69e562f8f332a38"} Jul 06 21:24:51.970532 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:51.970502 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd"] Jul 06 21:24:51.974053 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:51.973995 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd" Jul 06 21:24:51.976140 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:51.976117 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-577d057f-serving-cert\"" Jul 06 21:24:51.983112 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:51.983084 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:51.988284 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:51.988249 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd"] Jul 06 21:24:52.006259 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:52.005817 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" podStartSLOduration=2.005783568 podStartE2EDuration="2.005783568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:24:51.998935264 +0000 UTC m=+1576.317158525" watchObservedRunningTime="2026-07-06 21:24:52.005783568 +0000 UTC m=+1576.324006829" Jul 06 21:24:52.091700 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:52.091651 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95a4ace6-c881-480e-b1a4-797907d57e2b-maas-api-tls\") pod \"maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd\" (UID: \"95a4ace6-c881-480e-b1a4-797907d57e2b\") " pod="opendatahub/maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd" Jul 06 21:24:52.091700 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:52.091697 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x55rk\" (UniqueName: \"kubernetes.io/projected/95a4ace6-c881-480e-b1a4-797907d57e2b-kube-api-access-x55rk\") pod \"maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd\" (UID: \"95a4ace6-c881-480e-b1a4-797907d57e2b\") " pod="opendatahub/maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd" Jul 06 21:24:52.091985 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:52.091743 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95a4ace6-c881-480e-b1a4-797907d57e2b-openshift-service-ca\") pod \"maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd\" (UID: \"95a4ace6-c881-480e-b1a4-797907d57e2b\") " pod="opendatahub/maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd" Jul 06 21:24:52.192724 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:52.192687 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95a4ace6-c881-480e-b1a4-797907d57e2b-maas-api-tls\") pod \"maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd\" (UID: \"95a4ace6-c881-480e-b1a4-797907d57e2b\") " pod="opendatahub/maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd" Jul 06 21:24:52.192724 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:52.192728 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x55rk\" (UniqueName: \"kubernetes.io/projected/95a4ace6-c881-480e-b1a4-797907d57e2b-kube-api-access-x55rk\") pod \"maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd\" (UID: \"95a4ace6-c881-480e-b1a4-797907d57e2b\") " pod="opendatahub/maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd" Jul 06 21:24:52.192993 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:52.192780 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95a4ace6-c881-480e-b1a4-797907d57e2b-openshift-service-ca\") pod \"maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd\" (UID: \"95a4ace6-c881-480e-b1a4-797907d57e2b\") " pod="opendatahub/maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd" Jul 06 21:24:52.192993 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:24:52.192836 2577 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-577d057f-serving-cert: secret "maas-api-e2e-ait-577d057f-serving-cert" not found Jul 06 21:24:52.192993 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:24:52.192910 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95a4ace6-c881-480e-b1a4-797907d57e2b-maas-api-tls podName:95a4ace6-c881-480e-b1a4-797907d57e2b nodeName:}" failed. No retries permitted until 2026-07-06 21:24:52.692886842 +0000 UTC m=+1577.011110091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/95a4ace6-c881-480e-b1a4-797907d57e2b-maas-api-tls") pod "maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd" (UID: "95a4ace6-c881-480e-b1a4-797907d57e2b") : secret "maas-api-e2e-ait-577d057f-serving-cert" not found Jul 06 21:24:52.193565 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:52.193541 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95a4ace6-c881-480e-b1a4-797907d57e2b-openshift-service-ca\") pod \"maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd\" (UID: \"95a4ace6-c881-480e-b1a4-797907d57e2b\") " pod="opendatahub/maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd" Jul 06 21:24:52.202392 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:52.202352 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x55rk\" (UniqueName: \"kubernetes.io/projected/95a4ace6-c881-480e-b1a4-797907d57e2b-kube-api-access-x55rk\") pod \"maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd\" (UID: \"95a4ace6-c881-480e-b1a4-797907d57e2b\") " pod="opendatahub/maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd" Jul 06 21:24:52.698133 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:52.698088 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95a4ace6-c881-480e-b1a4-797907d57e2b-maas-api-tls\") pod \"maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd\" (UID: \"95a4ace6-c881-480e-b1a4-797907d57e2b\") " pod="opendatahub/maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd" Jul 06 21:24:52.700688 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:52.700660 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95a4ace6-c881-480e-b1a4-797907d57e2b-maas-api-tls\") pod \"maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd\" (UID: \"95a4ace6-c881-480e-b1a4-797907d57e2b\") " pod="opendatahub/maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd" Jul 06 21:24:52.890662 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:52.890618 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd" Jul 06 21:24:52.991801 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:52.991767 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:53.050802 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:53.050771 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd"] Jul 06 21:24:53.052968 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:24:53.052934 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a4ace6_c881_480e_b1a4_797907d57e2b.slice/crio-ea644ea149ba900826100a0882e94e8860e6d2918cab47e87705717749cc2f7a WatchSource:0}: Error finding container ea644ea149ba900826100a0882e94e8860e6d2918cab47e87705717749cc2f7a: Status 404 returned error can't find the container with id ea644ea149ba900826100a0882e94e8860e6d2918cab47e87705717749cc2f7a Jul 06 21:24:53.980473 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:53.980432 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd" event={"ID":"95a4ace6-c881-480e-b1a4-797907d57e2b","Type":"ContainerStarted","Data":"1ee4327d00f62508bcc05e3552f2824dd75acb7c969009259a2274be7764260b"} Jul 06 21:24:53.980692 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:53.980494 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd" event={"ID":"95a4ace6-c881-480e-b1a4-797907d57e2b","Type":"ContainerStarted","Data":"ea644ea149ba900826100a0882e94e8860e6d2918cab47e87705717749cc2f7a"} Jul 06 21:24:53.980759 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:53.980721 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd" Jul 06 21:24:53.980801 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:53.980765 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:53.981775 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:53.981752 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:24:54.003521 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:54.003469 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd" podStartSLOduration=2.660146729 podStartE2EDuration="3.003451032s" totalImagesPullingTime="343.304303ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:24:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:24:54.001732904 +0000 UTC m=+1578.319956163" watchObservedRunningTime="2026-07-06 21:24:54.003451032 +0000 UTC m=+1578.321674292" Jul 06 21:24:57.485288 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:57.485247 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd"] Jul 06 21:24:57.485722 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:57.485491 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd" podUID="95a4ace6-c881-480e-b1a4-797907d57e2b" containerName="maas-api" containerID="cri-o://1ee4327d00f62508bcc05e3552f2824dd75acb7c969009259a2274be7764260b" gracePeriod=30 Jul 06 21:24:57.781025 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:57.780989 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd" Jul 06 21:24:57.860799 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:57.860760 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x55rk\" (UniqueName: \"kubernetes.io/projected/95a4ace6-c881-480e-b1a4-797907d57e2b-kube-api-access-x55rk\") pod \"95a4ace6-c881-480e-b1a4-797907d57e2b\" (UID: \"95a4ace6-c881-480e-b1a4-797907d57e2b\") " Jul 06 21:24:57.860965 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:57.860844 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95a4ace6-c881-480e-b1a4-797907d57e2b-openshift-service-ca\") pod \"95a4ace6-c881-480e-b1a4-797907d57e2b\" (UID: \"95a4ace6-c881-480e-b1a4-797907d57e2b\") " Jul 06 21:24:57.861077 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:57.861049 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95a4ace6-c881-480e-b1a4-797907d57e2b-maas-api-tls\") pod \"95a4ace6-c881-480e-b1a4-797907d57e2b\" (UID: \"95a4ace6-c881-480e-b1a4-797907d57e2b\") " Jul 06 21:24:57.861204 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:57.861185 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95a4ace6-c881-480e-b1a4-797907d57e2b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "95a4ace6-c881-480e-b1a4-797907d57e2b" (UID: "95a4ace6-c881-480e-b1a4-797907d57e2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:24:57.861400 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:57.861377 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95a4ace6-c881-480e-b1a4-797907d57e2b-openshift-service-ca\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:24:57.863219 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:57.863181 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a4ace6-c881-480e-b1a4-797907d57e2b-kube-api-access-x55rk" (OuterVolumeSpecName: "kube-api-access-x55rk") pod "95a4ace6-c881-480e-b1a4-797907d57e2b" (UID: "95a4ace6-c881-480e-b1a4-797907d57e2b"). InnerVolumeSpecName "kube-api-access-x55rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:24:57.863219 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:57.863192 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95a4ace6-c881-480e-b1a4-797907d57e2b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "95a4ace6-c881-480e-b1a4-797907d57e2b" (UID: "95a4ace6-c881-480e-b1a4-797907d57e2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:24:57.962508 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:57.962457 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x55rk\" (UniqueName: \"kubernetes.io/projected/95a4ace6-c881-480e-b1a4-797907d57e2b-kube-api-access-x55rk\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:24:57.962508 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:57.962499 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95a4ace6-c881-480e-b1a4-797907d57e2b-maas-api-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:24:58.003301 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:58.003264 2577 generic.go:356] "Generic (PLEG): container finished" podID="95a4ace6-c881-480e-b1a4-797907d57e2b" containerID="1ee4327d00f62508bcc05e3552f2824dd75acb7c969009259a2274be7764260b" exitCode=0 Jul 06 21:24:58.003486 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:58.003328 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd" Jul 06 21:24:58.003486 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:58.003345 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd" event={"ID":"95a4ace6-c881-480e-b1a4-797907d57e2b","Type":"ContainerDied","Data":"1ee4327d00f62508bcc05e3552f2824dd75acb7c969009259a2274be7764260b"} Jul 06 21:24:58.003486 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:58.003424 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd" event={"ID":"95a4ace6-c881-480e-b1a4-797907d57e2b","Type":"ContainerDied","Data":"ea644ea149ba900826100a0882e94e8860e6d2918cab47e87705717749cc2f7a"} Jul 06 21:24:58.003486 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:58.003448 2577 scope.go:117] "RemoveContainer" containerID="1ee4327d00f62508bcc05e3552f2824dd75acb7c969009259a2274be7764260b" Jul 06 21:24:58.013832 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:58.013808 2577 scope.go:117] "RemoveContainer" containerID="1ee4327d00f62508bcc05e3552f2824dd75acb7c969009259a2274be7764260b" Jul 06 21:24:58.014616 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:24:58.014580 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ee4327d00f62508bcc05e3552f2824dd75acb7c969009259a2274be7764260b\": container with ID starting with 1ee4327d00f62508bcc05e3552f2824dd75acb7c969009259a2274be7764260b not found: ID does not exist" containerID="1ee4327d00f62508bcc05e3552f2824dd75acb7c969009259a2274be7764260b" Jul 06 21:24:58.014766 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:58.014630 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ee4327d00f62508bcc05e3552f2824dd75acb7c969009259a2274be7764260b"} err="failed to get container status \"1ee4327d00f62508bcc05e3552f2824dd75acb7c969009259a2274be7764260b\": rpc error: code = NotFound desc = could not find container \"1ee4327d00f62508bcc05e3552f2824dd75acb7c969009259a2274be7764260b\": container with ID starting with 1ee4327d00f62508bcc05e3552f2824dd75acb7c969009259a2274be7764260b not found: ID does not exist" Jul 06 21:24:58.028418 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:58.028340 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd"] Jul 06 21:24:58.032335 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:58.032288 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-577d057f-7bc6f74db5-xrgtd"] Jul 06 21:24:58.169320 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:24:58.169279 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95a4ace6-c881-480e-b1a4-797907d57e2b" path="/var/lib/kubelet/pods/95a4ace6-c881-480e-b1a4-797907d57e2b/volumes" Jul 06 21:25:03.034189 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:03.034089 2577 generic.go:356] "Generic (PLEG): container finished" podID="aabdb90e-9aff-4219-8270-de72d786149b" containerID="41c40ad5943d62a642349c2b93a2cea573a749d233f626496bf9415c0cb056a3" exitCode=0 Jul 06 21:25:03.034189 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:03.034161 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" event={"ID":"aabdb90e-9aff-4219-8270-de72d786149b","Type":"ContainerDied","Data":"41c40ad5943d62a642349c2b93a2cea573a749d233f626496bf9415c0cb056a3"} Jul 06 21:25:03.034669 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:03.034652 2577 scope.go:117] "RemoveContainer" containerID="41c40ad5943d62a642349c2b93a2cea573a749d233f626496bf9415c0cb056a3" Jul 06 21:25:03.035697 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:03.035666 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:25:03.035796 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:03.035752 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:25:03.035863 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:03.035818 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:25:03.213840 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:03.213797 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz"] Jul 06 21:25:04.041959 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:04.041911 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" event={"ID":"aabdb90e-9aff-4219-8270-de72d786149b","Type":"ContainerStarted","Data":"662bfc9082ce98c99bcfec1e09c0e31ec7ac9b5fe38831601cf557197f709335"} Jul 06 21:25:04.042400 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:04.041971 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" podUID="aabdb90e-9aff-4219-8270-de72d786149b" containerName="istio-proxy" containerID="cri-o://662bfc9082ce98c99bcfec1e09c0e31ec7ac9b5fe38831601cf557197f709335" gracePeriod=30 Jul 06 21:25:08.980944 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:08.980906 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:25:09.312360 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.312327 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:25:09.378313 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.378277 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aabdb90e-9aff-4219-8270-de72d786149b-workload-certs\") pod \"aabdb90e-9aff-4219-8270-de72d786149b\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " Jul 06 21:25:09.378313 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.378316 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aabdb90e-9aff-4219-8270-de72d786149b-istio-envoy\") pod \"aabdb90e-9aff-4219-8270-de72d786149b\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " Jul 06 21:25:09.378569 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.378332 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aabdb90e-9aff-4219-8270-de72d786149b-credential-socket\") pod \"aabdb90e-9aff-4219-8270-de72d786149b\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " Jul 06 21:25:09.378569 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.378370 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aabdb90e-9aff-4219-8270-de72d786149b-istio-data\") pod \"aabdb90e-9aff-4219-8270-de72d786149b\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " Jul 06 21:25:09.378569 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.378451 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aabdb90e-9aff-4219-8270-de72d786149b-istio-token\") pod \"aabdb90e-9aff-4219-8270-de72d786149b\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " Jul 06 21:25:09.378569 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.378494 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aabdb90e-9aff-4219-8270-de72d786149b-workload-socket\") pod \"aabdb90e-9aff-4219-8270-de72d786149b\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " Jul 06 21:25:09.378569 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.378527 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqctr\" (UniqueName: \"kubernetes.io/projected/aabdb90e-9aff-4219-8270-de72d786149b-kube-api-access-cqctr\") pod \"aabdb90e-9aff-4219-8270-de72d786149b\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " Jul 06 21:25:09.378569 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.378550 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aabdb90e-9aff-4219-8270-de72d786149b-istiod-ca-cert\") pod \"aabdb90e-9aff-4219-8270-de72d786149b\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " Jul 06 21:25:09.378871 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.378594 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aabdb90e-9aff-4219-8270-de72d786149b-istio-podinfo\") pod \"aabdb90e-9aff-4219-8270-de72d786149b\" (UID: \"aabdb90e-9aff-4219-8270-de72d786149b\") " Jul 06 21:25:09.378871 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.378641 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aabdb90e-9aff-4219-8270-de72d786149b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "aabdb90e-9aff-4219-8270-de72d786149b" (UID: "aabdb90e-9aff-4219-8270-de72d786149b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:25:09.378871 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.378697 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aabdb90e-9aff-4219-8270-de72d786149b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "aabdb90e-9aff-4219-8270-de72d786149b" (UID: "aabdb90e-9aff-4219-8270-de72d786149b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:25:09.379055 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.378877 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aabdb90e-9aff-4219-8270-de72d786149b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "aabdb90e-9aff-4219-8270-de72d786149b" (UID: "aabdb90e-9aff-4219-8270-de72d786149b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:25:09.379120 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.379070 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aabdb90e-9aff-4219-8270-de72d786149b-workload-socket\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:25:09.379120 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.379092 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aabdb90e-9aff-4219-8270-de72d786149b-workload-certs\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:25:09.379120 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.379092 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aabdb90e-9aff-4219-8270-de72d786149b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "aabdb90e-9aff-4219-8270-de72d786149b" (UID: "aabdb90e-9aff-4219-8270-de72d786149b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:25:09.379120 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.379106 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aabdb90e-9aff-4219-8270-de72d786149b-credential-socket\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:25:09.379701 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.379675 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aabdb90e-9aff-4219-8270-de72d786149b-istio-data" (OuterVolumeSpecName: "istio-data") pod "aabdb90e-9aff-4219-8270-de72d786149b" (UID: "aabdb90e-9aff-4219-8270-de72d786149b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:25:09.380717 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.380695 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aabdb90e-9aff-4219-8270-de72d786149b-kube-api-access-cqctr" (OuterVolumeSpecName: "kube-api-access-cqctr") pod "aabdb90e-9aff-4219-8270-de72d786149b" (UID: "aabdb90e-9aff-4219-8270-de72d786149b"). InnerVolumeSpecName "kube-api-access-cqctr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:25:09.380858 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.380841 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aabdb90e-9aff-4219-8270-de72d786149b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "aabdb90e-9aff-4219-8270-de72d786149b" (UID: "aabdb90e-9aff-4219-8270-de72d786149b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 21:25:09.380987 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.380968 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aabdb90e-9aff-4219-8270-de72d786149b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "aabdb90e-9aff-4219-8270-de72d786149b" (UID: "aabdb90e-9aff-4219-8270-de72d786149b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:25:09.380987 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.380976 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aabdb90e-9aff-4219-8270-de72d786149b-istio-token" (OuterVolumeSpecName: "istio-token") pod "aabdb90e-9aff-4219-8270-de72d786149b" (UID: "aabdb90e-9aff-4219-8270-de72d786149b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:25:09.480042 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.479964 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aabdb90e-9aff-4219-8270-de72d786149b-istio-envoy\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:25:09.480042 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.480001 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aabdb90e-9aff-4219-8270-de72d786149b-istio-data\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:25:09.480042 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.480042 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aabdb90e-9aff-4219-8270-de72d786149b-istio-token\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:25:09.480042 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.480051 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqctr\" (UniqueName: \"kubernetes.io/projected/aabdb90e-9aff-4219-8270-de72d786149b-kube-api-access-cqctr\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:25:09.480329 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.480060 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aabdb90e-9aff-4219-8270-de72d786149b-istiod-ca-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:25:09.480329 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.480069 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aabdb90e-9aff-4219-8270-de72d786149b-istio-podinfo\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:25:09.969120 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.969074 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm"] Jul 06 21:25:09.969742 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.969721 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95a4ace6-c881-480e-b1a4-797907d57e2b" containerName="maas-api" Jul 06 21:25:09.969793 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.969747 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="95a4ace6-c881-480e-b1a4-797907d57e2b" containerName="maas-api" Jul 06 21:25:09.969793 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.969776 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aabdb90e-9aff-4219-8270-de72d786149b" containerName="istio-proxy" Jul 06 21:25:09.969793 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.969786 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="aabdb90e-9aff-4219-8270-de72d786149b" containerName="istio-proxy" Jul 06 21:25:09.969902 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.969812 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aabdb90e-9aff-4219-8270-de72d786149b" containerName="istio-proxy" Jul 06 21:25:09.969902 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.969822 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="aabdb90e-9aff-4219-8270-de72d786149b" containerName="istio-proxy" Jul 06 21:25:09.969975 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.969934 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="aabdb90e-9aff-4219-8270-de72d786149b" containerName="istio-proxy" Jul 06 21:25:09.969975 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.969951 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="95a4ace6-c881-480e-b1a4-797907d57e2b" containerName="maas-api" Jul 06 21:25:09.969975 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.969962 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="aabdb90e-9aff-4219-8270-de72d786149b" containerName="istio-proxy" Jul 06 21:25:09.973782 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.973758 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:09.977380 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.977356 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-a1327588-openshift-default-dockercfg-4d7wv\"" Jul 06 21:25:09.988268 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:09.988235 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm"] Jul 06 21:25:10.081632 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.081591 2577 generic.go:356] "Generic (PLEG): container finished" podID="aabdb90e-9aff-4219-8270-de72d786149b" containerID="662bfc9082ce98c99bcfec1e09c0e31ec7ac9b5fe38831601cf557197f709335" exitCode=0 Jul 06 21:25:10.081839 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.081650 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" Jul 06 21:25:10.081839 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.081677 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" event={"ID":"aabdb90e-9aff-4219-8270-de72d786149b","Type":"ContainerDied","Data":"662bfc9082ce98c99bcfec1e09c0e31ec7ac9b5fe38831601cf557197f709335"} Jul 06 21:25:10.081839 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.081744 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz" event={"ID":"aabdb90e-9aff-4219-8270-de72d786149b","Type":"ContainerDied","Data":"0ed94421d43c9f9348396f6c58afaa415a5371d49faf2b03d69e562f8f332a38"} Jul 06 21:25:10.081839 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.081776 2577 scope.go:117] "RemoveContainer" containerID="662bfc9082ce98c99bcfec1e09c0e31ec7ac9b5fe38831601cf557197f709335" Jul 06 21:25:10.087329 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.086782 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-credential-socket\") pod \"e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:10.087329 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.086861 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9vtt\" (UniqueName: \"kubernetes.io/projected/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-kube-api-access-b9vtt\") pod \"e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:10.087329 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.086931 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-istio-envoy\") pod \"e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:10.087329 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.086973 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-istio-podinfo\") pod \"e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:10.087329 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.087047 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-workload-socket\") pod \"e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:10.087329 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.087098 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-istio-data\") pod \"e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:10.087329 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.087122 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-istio-token\") pod \"e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:10.087329 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.087146 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-istiod-ca-cert\") pod \"e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:10.087329 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.087200 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-workload-certs\") pod \"e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:10.092465 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.092445 2577 scope.go:117] "RemoveContainer" containerID="41c40ad5943d62a642349c2b93a2cea573a749d233f626496bf9415c0cb056a3" Jul 06 21:25:10.101493 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.101468 2577 scope.go:117] "RemoveContainer" containerID="662bfc9082ce98c99bcfec1e09c0e31ec7ac9b5fe38831601cf557197f709335" Jul 06 21:25:10.101806 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:25:10.101787 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"662bfc9082ce98c99bcfec1e09c0e31ec7ac9b5fe38831601cf557197f709335\": container with ID starting with 662bfc9082ce98c99bcfec1e09c0e31ec7ac9b5fe38831601cf557197f709335 not found: ID does not exist" containerID="662bfc9082ce98c99bcfec1e09c0e31ec7ac9b5fe38831601cf557197f709335" Jul 06 21:25:10.101861 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.101818 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"662bfc9082ce98c99bcfec1e09c0e31ec7ac9b5fe38831601cf557197f709335"} err="failed to get container status \"662bfc9082ce98c99bcfec1e09c0e31ec7ac9b5fe38831601cf557197f709335\": rpc error: code = NotFound desc = could not find container \"662bfc9082ce98c99bcfec1e09c0e31ec7ac9b5fe38831601cf557197f709335\": container with ID starting with 662bfc9082ce98c99bcfec1e09c0e31ec7ac9b5fe38831601cf557197f709335 not found: ID does not exist" Jul 06 21:25:10.101861 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.101836 2577 scope.go:117] "RemoveContainer" containerID="41c40ad5943d62a642349c2b93a2cea573a749d233f626496bf9415c0cb056a3" Jul 06 21:25:10.102207 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:25:10.102186 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41c40ad5943d62a642349c2b93a2cea573a749d233f626496bf9415c0cb056a3\": container with ID starting with 41c40ad5943d62a642349c2b93a2cea573a749d233f626496bf9415c0cb056a3 not found: ID does not exist" containerID="41c40ad5943d62a642349c2b93a2cea573a749d233f626496bf9415c0cb056a3" Jul 06 21:25:10.102271 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.102211 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41c40ad5943d62a642349c2b93a2cea573a749d233f626496bf9415c0cb056a3"} err="failed to get container status \"41c40ad5943d62a642349c2b93a2cea573a749d233f626496bf9415c0cb056a3\": rpc error: code = NotFound desc = could not find container \"41c40ad5943d62a642349c2b93a2cea573a749d233f626496bf9415c0cb056a3\": container with ID starting with 41c40ad5943d62a642349c2b93a2cea573a749d233f626496bf9415c0cb056a3 not found: ID does not exist" Jul 06 21:25:10.111060 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.110989 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz"] Jul 06 21:25:10.116601 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.116565 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-577d057f-openshift-default-8cc5ff965-5scvz"] Jul 06 21:25:10.169355 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.169311 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aabdb90e-9aff-4219-8270-de72d786149b" path="/var/lib/kubelet/pods/aabdb90e-9aff-4219-8270-de72d786149b/volumes" Jul 06 21:25:10.187784 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.187740 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-istio-envoy\") pod \"e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:10.187784 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.187796 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-istio-podinfo\") pod \"e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:10.188067 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.187938 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-workload-socket\") pod \"e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:10.188067 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.187987 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-istio-data\") pod \"e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:10.188067 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.188021 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-istio-token\") pod \"e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:10.188067 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.188044 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-istiod-ca-cert\") pod \"e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:10.188294 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.188088 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-workload-certs\") pod \"e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:10.188294 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.188158 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-credential-socket\") pod \"e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:10.188294 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.188218 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9vtt\" (UniqueName: \"kubernetes.io/projected/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-kube-api-access-b9vtt\") pod \"e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:10.188457 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.188387 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-workload-socket\") pod \"e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:10.188529 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.188480 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-istio-data\") pod \"e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:10.188628 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.188604 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-workload-certs\") pod \"e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:10.188691 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.188660 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-credential-socket\") pod \"e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:10.188931 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.188910 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-istiod-ca-cert\") pod \"e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:10.190168 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.190149 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-istio-envoy\") pod \"e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:10.190561 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.190538 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-istio-podinfo\") pod \"e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:10.197334 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.197298 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-istio-token\") pod \"e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:10.198195 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.198164 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9vtt\" (UniqueName: \"kubernetes.io/projected/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-kube-api-access-b9vtt\") pod \"e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:10.287279 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.287173 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:10.441056 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.441030 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm"] Jul 06 21:25:10.443067 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:25:10.443028 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2868bebe_a3cb_4e79_b1d4_81b4314fcd38.slice/crio-550ee828eff71dd7b21234ea917bc790a56561e9571f5b2ce270147edada886a WatchSource:0}: Error finding container 550ee828eff71dd7b21234ea917bc790a56561e9571f5b2ce270147edada886a: Status 404 returned error can't find the container with id 550ee828eff71dd7b21234ea917bc790a56561e9571f5b2ce270147edada886a Jul 06 21:25:10.445289 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.445256 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:25:10.445379 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.445344 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:25:10.445379 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:10.445374 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:25:11.088479 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:11.088443 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" event={"ID":"2868bebe-a3cb-4e79-b1d4-81b4314fcd38","Type":"ContainerStarted","Data":"94f13293adb90ba7d21e78d1e7efef6cd4c9c3631bfadda9b3a0925cc2139df5"} Jul 06 21:25:11.088964 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:11.088511 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" event={"ID":"2868bebe-a3cb-4e79-b1d4-81b4314fcd38","Type":"ContainerStarted","Data":"550ee828eff71dd7b21234ea917bc790a56561e9571f5b2ce270147edada886a"} Jul 06 21:25:11.118194 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:11.118126 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" podStartSLOduration=2.118105067 podStartE2EDuration="2.118105067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:25:11.116058966 +0000 UTC m=+1595.434282250" watchObservedRunningTime="2026-07-06 21:25:11.118105067 +0000 UTC m=+1595.436328328" Jul 06 21:25:11.285907 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:11.285860 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6"] Jul 06 21:25:11.289648 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:11.289618 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:11.289814 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:11.289734 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6" Jul 06 21:25:11.292505 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:11.292475 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-a1327588-serving-cert\"" Jul 06 21:25:11.292798 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:11.292778 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:11.301883 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:11.301850 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6"] Jul 06 21:25:11.402173 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:11.402128 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fx44\" (UniqueName: \"kubernetes.io/projected/7f0fb153-e6cc-497e-95e1-55f04f1534f3-kube-api-access-9fx44\") pod \"maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6\" (UID: \"7f0fb153-e6cc-497e-95e1-55f04f1534f3\") " pod="opendatahub/maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6" Jul 06 21:25:11.402544 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:11.402514 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f0fb153-e6cc-497e-95e1-55f04f1534f3-openshift-service-ca\") pod \"maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6\" (UID: \"7f0fb153-e6cc-497e-95e1-55f04f1534f3\") " pod="opendatahub/maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6" Jul 06 21:25:11.402679 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:11.402558 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f0fb153-e6cc-497e-95e1-55f04f1534f3-maas-api-tls\") pod \"maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6\" (UID: \"7f0fb153-e6cc-497e-95e1-55f04f1534f3\") " pod="opendatahub/maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6" Jul 06 21:25:11.504057 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:11.503998 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f0fb153-e6cc-497e-95e1-55f04f1534f3-openshift-service-ca\") pod \"maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6\" (UID: \"7f0fb153-e6cc-497e-95e1-55f04f1534f3\") " pod="opendatahub/maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6" Jul 06 21:25:11.504057 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:11.504059 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f0fb153-e6cc-497e-95e1-55f04f1534f3-maas-api-tls\") pod \"maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6\" (UID: \"7f0fb153-e6cc-497e-95e1-55f04f1534f3\") " pod="opendatahub/maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6" Jul 06 21:25:11.504342 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:11.504148 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fx44\" (UniqueName: \"kubernetes.io/projected/7f0fb153-e6cc-497e-95e1-55f04f1534f3-kube-api-access-9fx44\") pod \"maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6\" (UID: \"7f0fb153-e6cc-497e-95e1-55f04f1534f3\") " pod="opendatahub/maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6" Jul 06 21:25:11.504342 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:25:11.504244 2577 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-a1327588-serving-cert: secret "maas-api-e2e-derive-a1327588-serving-cert" not found Jul 06 21:25:11.504342 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:25:11.504329 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f0fb153-e6cc-497e-95e1-55f04f1534f3-maas-api-tls podName:7f0fb153-e6cc-497e-95e1-55f04f1534f3 nodeName:}" failed. No retries permitted until 2026-07-06 21:25:12.00430492 +0000 UTC m=+1596.322528164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7f0fb153-e6cc-497e-95e1-55f04f1534f3-maas-api-tls") pod "maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6" (UID: "7f0fb153-e6cc-497e-95e1-55f04f1534f3") : secret "maas-api-e2e-derive-a1327588-serving-cert" not found Jul 06 21:25:11.504780 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:11.504755 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f0fb153-e6cc-497e-95e1-55f04f1534f3-openshift-service-ca\") pod \"maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6\" (UID: \"7f0fb153-e6cc-497e-95e1-55f04f1534f3\") " pod="opendatahub/maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6" Jul 06 21:25:11.514420 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:11.514375 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fx44\" (UniqueName: \"kubernetes.io/projected/7f0fb153-e6cc-497e-95e1-55f04f1534f3-kube-api-access-9fx44\") pod \"maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6\" (UID: \"7f0fb153-e6cc-497e-95e1-55f04f1534f3\") " pod="opendatahub/maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6" Jul 06 21:25:12.009719 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:12.009673 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f0fb153-e6cc-497e-95e1-55f04f1534f3-maas-api-tls\") pod \"maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6\" (UID: \"7f0fb153-e6cc-497e-95e1-55f04f1534f3\") " pod="opendatahub/maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6" Jul 06 21:25:12.012279 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:12.012253 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f0fb153-e6cc-497e-95e1-55f04f1534f3-maas-api-tls\") pod \"maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6\" (UID: \"7f0fb153-e6cc-497e-95e1-55f04f1534f3\") " pod="opendatahub/maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6" Jul 06 21:25:12.094590 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:12.094555 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:12.095873 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:12.095845 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:12.206436 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:12.206404 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6" Jul 06 21:25:12.359471 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:12.359444 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6"] Jul 06 21:25:12.362241 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:25:12.362212 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f0fb153_e6cc_497e_95e1_55f04f1534f3.slice/crio-166a67c2634fd89d458b7c50b9a38368b30fdd101d6c1134c937c148de42ebf3 WatchSource:0}: Error finding container 166a67c2634fd89d458b7c50b9a38368b30fdd101d6c1134c937c148de42ebf3: Status 404 returned error can't find the container with id 166a67c2634fd89d458b7c50b9a38368b30fdd101d6c1134c937c148de42ebf3 Jul 06 21:25:13.100725 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:13.100670 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6" event={"ID":"7f0fb153-e6cc-497e-95e1-55f04f1534f3","Type":"ContainerStarted","Data":"011a7794c643cf57df04f50b19c034f4eaf40d3bccd5859e1c2d88101954a58a"} Jul 06 21:25:13.101249 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:13.100763 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6" event={"ID":"7f0fb153-e6cc-497e-95e1-55f04f1534f3","Type":"ContainerStarted","Data":"166a67c2634fd89d458b7c50b9a38368b30fdd101d6c1134c937c148de42ebf3"} Jul 06 21:25:13.101249 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:13.100977 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6" Jul 06 21:25:13.121997 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:13.121943 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6" podStartSLOduration=1.824266595 podStartE2EDuration="2.121927477s" totalImagesPullingTime="297.660882ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:25:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:25:13.120782661 +0000 UTC m=+1597.439005922" watchObservedRunningTime="2026-07-06 21:25:13.121927477 +0000 UTC m=+1597.440150736" Jul 06 21:25:15.759296 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:15.759245 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm"] Jul 06 21:25:15.759747 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:15.759476 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" podUID="2868bebe-a3cb-4e79-b1d4-81b4314fcd38" containerName="istio-proxy" containerID="cri-o://94f13293adb90ba7d21e78d1e7efef6cd4c9c3631bfadda9b3a0925cc2139df5" gracePeriod=30 Jul 06 21:25:15.960325 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:15.960287 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6"] Jul 06 21:25:15.961747 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:15.960796 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6" podUID="7f0fb153-e6cc-497e-95e1-55f04f1534f3" containerName="maas-api" containerID="cri-o://011a7794c643cf57df04f50b19c034f4eaf40d3bccd5859e1c2d88101954a58a" gracePeriod=30 Jul 06 21:25:16.119264 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:16.119232 2577 generic.go:356] "Generic (PLEG): container finished" podID="7f0fb153-e6cc-497e-95e1-55f04f1534f3" containerID="011a7794c643cf57df04f50b19c034f4eaf40d3bccd5859e1c2d88101954a58a" exitCode=0 Jul 06 21:25:16.119424 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:16.119301 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6" event={"ID":"7f0fb153-e6cc-497e-95e1-55f04f1534f3","Type":"ContainerDied","Data":"011a7794c643cf57df04f50b19c034f4eaf40d3bccd5859e1c2d88101954a58a"} Jul 06 21:25:16.262535 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:16.262506 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6" Jul 06 21:25:16.354159 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:16.354053 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fx44\" (UniqueName: \"kubernetes.io/projected/7f0fb153-e6cc-497e-95e1-55f04f1534f3-kube-api-access-9fx44\") pod \"7f0fb153-e6cc-497e-95e1-55f04f1534f3\" (UID: \"7f0fb153-e6cc-497e-95e1-55f04f1534f3\") " Jul 06 21:25:16.354339 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:16.354166 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f0fb153-e6cc-497e-95e1-55f04f1534f3-maas-api-tls\") pod \"7f0fb153-e6cc-497e-95e1-55f04f1534f3\" (UID: \"7f0fb153-e6cc-497e-95e1-55f04f1534f3\") " Jul 06 21:25:16.354339 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:16.354214 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f0fb153-e6cc-497e-95e1-55f04f1534f3-openshift-service-ca\") pod \"7f0fb153-e6cc-497e-95e1-55f04f1534f3\" (UID: \"7f0fb153-e6cc-497e-95e1-55f04f1534f3\") " Jul 06 21:25:16.354608 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:16.354581 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f0fb153-e6cc-497e-95e1-55f04f1534f3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7f0fb153-e6cc-497e-95e1-55f04f1534f3" (UID: "7f0fb153-e6cc-497e-95e1-55f04f1534f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:25:16.356374 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:16.356348 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f0fb153-e6cc-497e-95e1-55f04f1534f3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7f0fb153-e6cc-497e-95e1-55f04f1534f3" (UID: "7f0fb153-e6cc-497e-95e1-55f04f1534f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:25:16.356447 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:16.356386 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0fb153-e6cc-497e-95e1-55f04f1534f3-kube-api-access-9fx44" (OuterVolumeSpecName: "kube-api-access-9fx44") pod "7f0fb153-e6cc-497e-95e1-55f04f1534f3" (UID: "7f0fb153-e6cc-497e-95e1-55f04f1534f3"). InnerVolumeSpecName "kube-api-access-9fx44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:25:16.455260 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:16.455200 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fx44\" (UniqueName: \"kubernetes.io/projected/7f0fb153-e6cc-497e-95e1-55f04f1534f3-kube-api-access-9fx44\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:25:16.455260 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:16.455253 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f0fb153-e6cc-497e-95e1-55f04f1534f3-maas-api-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:25:16.455260 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:16.455267 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f0fb153-e6cc-497e-95e1-55f04f1534f3-openshift-service-ca\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:25:17.128661 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:17.128611 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6" event={"ID":"7f0fb153-e6cc-497e-95e1-55f04f1534f3","Type":"ContainerDied","Data":"166a67c2634fd89d458b7c50b9a38368b30fdd101d6c1134c937c148de42ebf3"} Jul 06 21:25:17.129182 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:17.128699 2577 scope.go:117] "RemoveContainer" containerID="011a7794c643cf57df04f50b19c034f4eaf40d3bccd5859e1c2d88101954a58a" Jul 06 21:25:17.129182 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:17.128864 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6" Jul 06 21:25:17.184568 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:17.184535 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6"] Jul 06 21:25:17.189621 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:17.189579 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-a1327588-6b9894bc8d-sp6s6"] Jul 06 21:25:18.168837 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:18.168796 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f0fb153-e6cc-497e-95e1-55f04f1534f3" path="/var/lib/kubelet/pods/7f0fb153-e6cc-497e-95e1-55f04f1534f3/volumes" Jul 06 21:25:21.012390 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.012352 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:21.101172 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.101079 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-istiod-ca-cert\") pod \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " Jul 06 21:25:21.101172 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.101146 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-istio-envoy\") pod \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " Jul 06 21:25:21.101398 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.101184 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9vtt\" (UniqueName: \"kubernetes.io/projected/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-kube-api-access-b9vtt\") pod \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " Jul 06 21:25:21.101398 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.101222 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-istio-data\") pod \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " Jul 06 21:25:21.101398 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.101246 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-credential-socket\") pod \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " Jul 06 21:25:21.101398 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.101276 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-istio-podinfo\") pod \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " Jul 06 21:25:21.101398 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.101312 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-workload-socket\") pod \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " Jul 06 21:25:21.101398 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.101361 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-istio-token\") pod \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " Jul 06 21:25:21.101705 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.101403 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-workload-certs\") pod \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\" (UID: \"2868bebe-a3cb-4e79-b1d4-81b4314fcd38\") " Jul 06 21:25:21.101705 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.101573 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2868bebe-a3cb-4e79-b1d4-81b4314fcd38" (UID: "2868bebe-a3cb-4e79-b1d4-81b4314fcd38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:25:21.101819 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.101785 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2868bebe-a3cb-4e79-b1d4-81b4314fcd38" (UID: "2868bebe-a3cb-4e79-b1d4-81b4314fcd38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:25:21.101819 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.101803 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-istiod-ca-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:25:21.102054 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.101951 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2868bebe-a3cb-4e79-b1d4-81b4314fcd38" (UID: "2868bebe-a3cb-4e79-b1d4-81b4314fcd38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:25:21.102054 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.101984 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2868bebe-a3cb-4e79-b1d4-81b4314fcd38" (UID: "2868bebe-a3cb-4e79-b1d4-81b4314fcd38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:25:21.102210 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.102120 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-istio-data" (OuterVolumeSpecName: "istio-data") pod "2868bebe-a3cb-4e79-b1d4-81b4314fcd38" (UID: "2868bebe-a3cb-4e79-b1d4-81b4314fcd38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:25:21.103721 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.103681 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-istio-token" (OuterVolumeSpecName: "istio-token") pod "2868bebe-a3cb-4e79-b1d4-81b4314fcd38" (UID: "2868bebe-a3cb-4e79-b1d4-81b4314fcd38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:25:21.104041 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.103988 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2868bebe-a3cb-4e79-b1d4-81b4314fcd38" (UID: "2868bebe-a3cb-4e79-b1d4-81b4314fcd38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:25:21.104141 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.104095 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-kube-api-access-b9vtt" (OuterVolumeSpecName: "kube-api-access-b9vtt") pod "2868bebe-a3cb-4e79-b1d4-81b4314fcd38" (UID: "2868bebe-a3cb-4e79-b1d4-81b4314fcd38"). InnerVolumeSpecName "kube-api-access-b9vtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:25:21.104196 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.104135 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2868bebe-a3cb-4e79-b1d4-81b4314fcd38" (UID: "2868bebe-a3cb-4e79-b1d4-81b4314fcd38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 21:25:21.158358 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.158316 2577 generic.go:356] "Generic (PLEG): container finished" podID="2868bebe-a3cb-4e79-b1d4-81b4314fcd38" containerID="94f13293adb90ba7d21e78d1e7efef6cd4c9c3631bfadda9b3a0925cc2139df5" exitCode=0 Jul 06 21:25:21.158358 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.158362 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" Jul 06 21:25:21.158619 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.158398 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" event={"ID":"2868bebe-a3cb-4e79-b1d4-81b4314fcd38","Type":"ContainerDied","Data":"94f13293adb90ba7d21e78d1e7efef6cd4c9c3631bfadda9b3a0925cc2139df5"} Jul 06 21:25:21.158619 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.158455 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm" event={"ID":"2868bebe-a3cb-4e79-b1d4-81b4314fcd38","Type":"ContainerDied","Data":"550ee828eff71dd7b21234ea917bc790a56561e9571f5b2ce270147edada886a"} Jul 06 21:25:21.158619 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.158477 2577 scope.go:117] "RemoveContainer" containerID="94f13293adb90ba7d21e78d1e7efef6cd4c9c3631bfadda9b3a0925cc2139df5" Jul 06 21:25:21.171402 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.171381 2577 scope.go:117] "RemoveContainer" containerID="94f13293adb90ba7d21e78d1e7efef6cd4c9c3631bfadda9b3a0925cc2139df5" Jul 06 21:25:21.171809 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:25:21.171786 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94f13293adb90ba7d21e78d1e7efef6cd4c9c3631bfadda9b3a0925cc2139df5\": container with ID starting with 94f13293adb90ba7d21e78d1e7efef6cd4c9c3631bfadda9b3a0925cc2139df5 not found: ID does not exist" containerID="94f13293adb90ba7d21e78d1e7efef6cd4c9c3631bfadda9b3a0925cc2139df5" Jul 06 21:25:21.171880 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.171820 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94f13293adb90ba7d21e78d1e7efef6cd4c9c3631bfadda9b3a0925cc2139df5"} err="failed to get container status \"94f13293adb90ba7d21e78d1e7efef6cd4c9c3631bfadda9b3a0925cc2139df5\": rpc error: code = NotFound desc = could not find container \"94f13293adb90ba7d21e78d1e7efef6cd4c9c3631bfadda9b3a0925cc2139df5\": container with ID starting with 94f13293adb90ba7d21e78d1e7efef6cd4c9c3631bfadda9b3a0925cc2139df5 not found: ID does not exist" Jul 06 21:25:21.183990 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:25:21.183955 2577 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2868bebe_a3cb_4e79_b1d4_81b4314fcd38.slice\": RecentStats: unable to find data in memory cache]" Jul 06 21:25:21.188711 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.188676 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm"] Jul 06 21:25:21.190901 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.190871 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-a1327588-openshift-default-6ff989b85f-xdmqm"] Jul 06 21:25:21.203338 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.203299 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-istio-data\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:25:21.203338 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.203328 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-credential-socket\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:25:21.203338 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.203339 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-istio-podinfo\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:25:21.203338 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.203349 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-workload-socket\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:25:21.203604 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.203358 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-istio-token\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:25:21.203604 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.203368 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-workload-certs\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:25:21.203604 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.203376 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-istio-envoy\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:25:21.203604 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:21.203386 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9vtt\" (UniqueName: \"kubernetes.io/projected/2868bebe-a3cb-4e79-b1d4-81b4314fcd38-kube-api-access-b9vtt\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:25:22.169071 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:25:22.169023 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2868bebe-a3cb-4e79-b1d4-81b4314fcd38" path="/var/lib/kubelet/pods/2868bebe-a3cb-4e79-b1d4-81b4314fcd38/volumes" Jul 06 21:27:00.812943 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.812855 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp"] Jul 06 21:27:00.813560 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.813323 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f0fb153-e6cc-497e-95e1-55f04f1534f3" containerName="maas-api" Jul 06 21:27:00.813560 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.813338 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f0fb153-e6cc-497e-95e1-55f04f1534f3" containerName="maas-api" Jul 06 21:27:00.813560 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.813358 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2868bebe-a3cb-4e79-b1d4-81b4314fcd38" containerName="istio-proxy" Jul 06 21:27:00.813560 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.813363 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2868bebe-a3cb-4e79-b1d4-81b4314fcd38" containerName="istio-proxy" Jul 06 21:27:00.813560 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.813439 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f0fb153-e6cc-497e-95e1-55f04f1534f3" containerName="maas-api" Jul 06 21:27:00.813560 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.813448 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2868bebe-a3cb-4e79-b1d4-81b4314fcd38" containerName="istio-proxy" Jul 06 21:27:00.816890 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.816868 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:00.819378 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.819345 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ef8479e9-openshift-default-dockercfg-2w22q\"" Jul 06 21:27:00.840427 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.840390 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp"] Jul 06 21:27:00.869891 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.869841 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58b06de-c3ee-4284-80ee-9e5c11a89a15-workload-certs\") pod \"e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:00.869891 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.869897 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58b06de-c3ee-4284-80ee-9e5c11a89a15-istio-podinfo\") pod \"e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:00.870181 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.869963 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbpg6\" (UniqueName: \"kubernetes.io/projected/e58b06de-c3ee-4284-80ee-9e5c11a89a15-kube-api-access-jbpg6\") pod \"e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:00.870181 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.870033 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58b06de-c3ee-4284-80ee-9e5c11a89a15-istiod-ca-cert\") pod \"e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:00.870181 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.870113 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58b06de-c3ee-4284-80ee-9e5c11a89a15-istio-data\") pod \"e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:00.870354 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.870181 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58b06de-c3ee-4284-80ee-9e5c11a89a15-istio-token\") pod \"e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:00.870354 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.870233 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58b06de-c3ee-4284-80ee-9e5c11a89a15-workload-socket\") pod \"e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:00.870354 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.870283 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58b06de-c3ee-4284-80ee-9e5c11a89a15-credential-socket\") pod \"e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:00.870496 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.870363 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58b06de-c3ee-4284-80ee-9e5c11a89a15-istio-envoy\") pod \"e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:00.971906 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.971867 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58b06de-c3ee-4284-80ee-9e5c11a89a15-istio-envoy\") pod \"e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:00.971906 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.971910 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58b06de-c3ee-4284-80ee-9e5c11a89a15-workload-certs\") pod \"e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:00.972184 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.972025 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58b06de-c3ee-4284-80ee-9e5c11a89a15-istio-podinfo\") pod \"e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:00.972184 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.972053 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbpg6\" (UniqueName: \"kubernetes.io/projected/e58b06de-c3ee-4284-80ee-9e5c11a89a15-kube-api-access-jbpg6\") pod \"e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:00.972184 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.972086 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58b06de-c3ee-4284-80ee-9e5c11a89a15-istiod-ca-cert\") pod \"e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:00.972184 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.972111 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58b06de-c3ee-4284-80ee-9e5c11a89a15-istio-data\") pod \"e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:00.972410 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.972316 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58b06de-c3ee-4284-80ee-9e5c11a89a15-istio-token\") pod \"e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:00.972468 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.972405 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58b06de-c3ee-4284-80ee-9e5c11a89a15-workload-socket\") pod \"e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:00.972468 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.972447 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58b06de-c3ee-4284-80ee-9e5c11a89a15-workload-certs\") pod \"e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:00.972571 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.972442 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58b06de-c3ee-4284-80ee-9e5c11a89a15-credential-socket\") pod \"e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:00.972571 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.972503 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58b06de-c3ee-4284-80ee-9e5c11a89a15-istio-data\") pod \"e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:00.972867 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.972827 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58b06de-c3ee-4284-80ee-9e5c11a89a15-workload-socket\") pod \"e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:00.972975 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.972871 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58b06de-c3ee-4284-80ee-9e5c11a89a15-credential-socket\") pod \"e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:00.973168 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.973149 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58b06de-c3ee-4284-80ee-9e5c11a89a15-istiod-ca-cert\") pod \"e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:00.974377 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.974350 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58b06de-c3ee-4284-80ee-9e5c11a89a15-istio-envoy\") pod \"e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:00.974636 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.974619 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58b06de-c3ee-4284-80ee-9e5c11a89a15-istio-podinfo\") pod \"e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:00.980605 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.980576 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58b06de-c3ee-4284-80ee-9e5c11a89a15-istio-token\") pod \"e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:00.980740 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:00.980711 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbpg6\" (UniqueName: \"kubernetes.io/projected/e58b06de-c3ee-4284-80ee-9e5c11a89a15-kube-api-access-jbpg6\") pod \"e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:01.129068 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:01.129021 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:01.284639 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:01.284613 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp"] Jul 06 21:27:01.286535 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:27:01.286506 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode58b06de_c3ee_4284_80ee_9e5c11a89a15.slice/crio-1da9373e070a4de295eccfdb9dce6cfd514361968273011bef1d9b5d85106870 WatchSource:0}: Error finding container 1da9373e070a4de295eccfdb9dce6cfd514361968273011bef1d9b5d85106870: Status 404 returned error can't find the container with id 1da9373e070a4de295eccfdb9dce6cfd514361968273011bef1d9b5d85106870 Jul 06 21:27:01.289098 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:01.289062 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:27:01.289174 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:01.289127 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:27:01.289174 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:01.289156 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:27:01.667731 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:01.667694 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" event={"ID":"e58b06de-c3ee-4284-80ee-9e5c11a89a15","Type":"ContainerStarted","Data":"099effe51bbf4476d0884a75f682e8119ab70fcc67dc9ad1c87b9dc5749c3e6b"} Jul 06 21:27:01.667934 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:01.667760 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" event={"ID":"e58b06de-c3ee-4284-80ee-9e5c11a89a15","Type":"ContainerStarted","Data":"1da9373e070a4de295eccfdb9dce6cfd514361968273011bef1d9b5d85106870"} Jul 06 21:27:01.701658 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:01.701595 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" podStartSLOduration=1.701575332 podStartE2EDuration="1.701575332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:27:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:27:01.698470043 +0000 UTC m=+1706.016693304" watchObservedRunningTime="2026-07-06 21:27:01.701575332 +0000 UTC m=+1706.019798592" Jul 06 21:27:02.129407 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:02.129361 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:02.131030 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:02.130990 2577 patch_prober.go:28] interesting pod/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 06 21:27:02.131168 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:02.131069 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" podUID="e58b06de-c3ee-4284-80ee-9e5c11a89a15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 06 21:27:03.134055 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:03.133998 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:03.677940 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:03.677893 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:03.679095 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:03.679073 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:03.730538 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:03.730502 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-ef8479e9-59f87c5584-264d5"] Jul 06 21:27:03.734349 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:03.734318 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-ef8479e9-59f87c5584-264d5" Jul 06 21:27:03.736779 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:03.736753 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-ef8479e9-serving-cert\"" Jul 06 21:27:03.747933 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:03.747891 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-ef8479e9-59f87c5584-264d5"] Jul 06 21:27:03.807089 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:03.807039 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6cc3890-cacb-4678-9ee5-c5611e5db8f4-openshift-service-ca\") pod \"maas-api-e2e-mt-ef8479e9-59f87c5584-264d5\" (UID: \"f6cc3890-cacb-4678-9ee5-c5611e5db8f4\") " pod="opendatahub/maas-api-e2e-mt-ef8479e9-59f87c5584-264d5" Jul 06 21:27:03.807286 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:03.807099 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6cc3890-cacb-4678-9ee5-c5611e5db8f4-maas-api-tls\") pod \"maas-api-e2e-mt-ef8479e9-59f87c5584-264d5\" (UID: \"f6cc3890-cacb-4678-9ee5-c5611e5db8f4\") " pod="opendatahub/maas-api-e2e-mt-ef8479e9-59f87c5584-264d5" Jul 06 21:27:03.807286 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:03.807246 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tg7p\" (UniqueName: \"kubernetes.io/projected/f6cc3890-cacb-4678-9ee5-c5611e5db8f4-kube-api-access-4tg7p\") pod \"maas-api-e2e-mt-ef8479e9-59f87c5584-264d5\" (UID: \"f6cc3890-cacb-4678-9ee5-c5611e5db8f4\") " pod="opendatahub/maas-api-e2e-mt-ef8479e9-59f87c5584-264d5" Jul 06 21:27:03.908796 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:03.908750 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6cc3890-cacb-4678-9ee5-c5611e5db8f4-openshift-service-ca\") pod \"maas-api-e2e-mt-ef8479e9-59f87c5584-264d5\" (UID: \"f6cc3890-cacb-4678-9ee5-c5611e5db8f4\") " pod="opendatahub/maas-api-e2e-mt-ef8479e9-59f87c5584-264d5" Jul 06 21:27:03.908796 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:03.908797 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6cc3890-cacb-4678-9ee5-c5611e5db8f4-maas-api-tls\") pod \"maas-api-e2e-mt-ef8479e9-59f87c5584-264d5\" (UID: \"f6cc3890-cacb-4678-9ee5-c5611e5db8f4\") " pod="opendatahub/maas-api-e2e-mt-ef8479e9-59f87c5584-264d5" Jul 06 21:27:03.909043 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:03.908877 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tg7p\" (UniqueName: \"kubernetes.io/projected/f6cc3890-cacb-4678-9ee5-c5611e5db8f4-kube-api-access-4tg7p\") pod \"maas-api-e2e-mt-ef8479e9-59f87c5584-264d5\" (UID: \"f6cc3890-cacb-4678-9ee5-c5611e5db8f4\") " pod="opendatahub/maas-api-e2e-mt-ef8479e9-59f87c5584-264d5" Jul 06 21:27:03.909626 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:03.909604 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6cc3890-cacb-4678-9ee5-c5611e5db8f4-openshift-service-ca\") pod \"maas-api-e2e-mt-ef8479e9-59f87c5584-264d5\" (UID: \"f6cc3890-cacb-4678-9ee5-c5611e5db8f4\") " pod="opendatahub/maas-api-e2e-mt-ef8479e9-59f87c5584-264d5" Jul 06 21:27:03.911716 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:03.911682 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6cc3890-cacb-4678-9ee5-c5611e5db8f4-maas-api-tls\") pod \"maas-api-e2e-mt-ef8479e9-59f87c5584-264d5\" (UID: \"f6cc3890-cacb-4678-9ee5-c5611e5db8f4\") " pod="opendatahub/maas-api-e2e-mt-ef8479e9-59f87c5584-264d5" Jul 06 21:27:03.919704 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:03.919675 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tg7p\" (UniqueName: \"kubernetes.io/projected/f6cc3890-cacb-4678-9ee5-c5611e5db8f4-kube-api-access-4tg7p\") pod \"maas-api-e2e-mt-ef8479e9-59f87c5584-264d5\" (UID: \"f6cc3890-cacb-4678-9ee5-c5611e5db8f4\") " pod="opendatahub/maas-api-e2e-mt-ef8479e9-59f87c5584-264d5" Jul 06 21:27:04.048855 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:04.048758 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-ef8479e9-59f87c5584-264d5" Jul 06 21:27:04.197378 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:04.197344 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-ef8479e9-59f87c5584-264d5"] Jul 06 21:27:04.200850 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:27:04.200811 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6cc3890_cacb_4678_9ee5_c5611e5db8f4.slice/crio-bc6e828f56b7826b0909b984356f1381052cde44c4969adc5db59cd8573924b4 WatchSource:0}: Error finding container bc6e828f56b7826b0909b984356f1381052cde44c4969adc5db59cd8573924b4: Status 404 returned error can't find the container with id bc6e828f56b7826b0909b984356f1381052cde44c4969adc5db59cd8573924b4 Jul 06 21:27:04.684548 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:04.684504 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-ef8479e9-59f87c5584-264d5" event={"ID":"f6cc3890-cacb-4678-9ee5-c5611e5db8f4","Type":"ContainerStarted","Data":"bc6e828f56b7826b0909b984356f1381052cde44c4969adc5db59cd8573924b4"} Jul 06 21:27:05.692128 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:05.692078 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-ef8479e9-59f87c5584-264d5" event={"ID":"f6cc3890-cacb-4678-9ee5-c5611e5db8f4","Type":"ContainerStarted","Data":"a66bbcbba319eaf4a05026ff73294cf2d6eb519cd4068ecdd5a01b6a1212269b"} Jul 06 21:27:05.692620 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:05.692171 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-ef8479e9-59f87c5584-264d5" Jul 06 21:27:05.710222 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:05.710167 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-ef8479e9-59f87c5584-264d5" podStartSLOduration=2.171307355 podStartE2EDuration="2.710151255s" totalImagesPullingTime="538.8439ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:27:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:27:05.708839335 +0000 UTC m=+1710.027062609" watchObservedRunningTime="2026-07-06 21:27:05.710151255 +0000 UTC m=+1710.028374515" Jul 06 21:27:11.702528 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:11.702500 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-ef8479e9-59f87c5584-264d5" Jul 06 21:27:16.062271 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:16.062225 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-ef8479e9-59f87c5584-264d5"] Jul 06 21:27:16.063140 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:16.063077 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-ef8479e9-59f87c5584-264d5" podUID="f6cc3890-cacb-4678-9ee5-c5611e5db8f4" containerName="maas-api" containerID="cri-o://a66bbcbba319eaf4a05026ff73294cf2d6eb519cd4068ecdd5a01b6a1212269b" gracePeriod=30 Jul 06 21:27:16.327791 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:16.327714 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-ef8479e9-59f87c5584-264d5" Jul 06 21:27:16.439869 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:16.439822 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6cc3890-cacb-4678-9ee5-c5611e5db8f4-maas-api-tls\") pod \"f6cc3890-cacb-4678-9ee5-c5611e5db8f4\" (UID: \"f6cc3890-cacb-4678-9ee5-c5611e5db8f4\") " Jul 06 21:27:16.440080 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:16.439898 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tg7p\" (UniqueName: \"kubernetes.io/projected/f6cc3890-cacb-4678-9ee5-c5611e5db8f4-kube-api-access-4tg7p\") pod \"f6cc3890-cacb-4678-9ee5-c5611e5db8f4\" (UID: \"f6cc3890-cacb-4678-9ee5-c5611e5db8f4\") " Jul 06 21:27:16.440080 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:16.439920 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6cc3890-cacb-4678-9ee5-c5611e5db8f4-openshift-service-ca\") pod \"f6cc3890-cacb-4678-9ee5-c5611e5db8f4\" (UID: \"f6cc3890-cacb-4678-9ee5-c5611e5db8f4\") " Jul 06 21:27:16.440376 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:16.440351 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6cc3890-cacb-4678-9ee5-c5611e5db8f4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f6cc3890-cacb-4678-9ee5-c5611e5db8f4" (UID: "f6cc3890-cacb-4678-9ee5-c5611e5db8f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:27:16.442300 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:16.442256 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6cc3890-cacb-4678-9ee5-c5611e5db8f4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f6cc3890-cacb-4678-9ee5-c5611e5db8f4" (UID: "f6cc3890-cacb-4678-9ee5-c5611e5db8f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:27:16.442300 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:16.442254 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6cc3890-cacb-4678-9ee5-c5611e5db8f4-kube-api-access-4tg7p" (OuterVolumeSpecName: "kube-api-access-4tg7p") pod "f6cc3890-cacb-4678-9ee5-c5611e5db8f4" (UID: "f6cc3890-cacb-4678-9ee5-c5611e5db8f4"). InnerVolumeSpecName "kube-api-access-4tg7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:27:16.541271 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:16.541228 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6cc3890-cacb-4678-9ee5-c5611e5db8f4-maas-api-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:27:16.541271 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:16.541265 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tg7p\" (UniqueName: \"kubernetes.io/projected/f6cc3890-cacb-4678-9ee5-c5611e5db8f4-kube-api-access-4tg7p\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:27:16.541271 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:16.541275 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6cc3890-cacb-4678-9ee5-c5611e5db8f4-openshift-service-ca\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:27:16.765795 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:16.765760 2577 generic.go:356] "Generic (PLEG): container finished" podID="f6cc3890-cacb-4678-9ee5-c5611e5db8f4" containerID="a66bbcbba319eaf4a05026ff73294cf2d6eb519cd4068ecdd5a01b6a1212269b" exitCode=0 Jul 06 21:27:16.766053 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:16.765832 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-ef8479e9-59f87c5584-264d5" Jul 06 21:27:16.766053 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:16.765853 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-ef8479e9-59f87c5584-264d5" event={"ID":"f6cc3890-cacb-4678-9ee5-c5611e5db8f4","Type":"ContainerDied","Data":"a66bbcbba319eaf4a05026ff73294cf2d6eb519cd4068ecdd5a01b6a1212269b"} Jul 06 21:27:16.766053 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:16.765926 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-ef8479e9-59f87c5584-264d5" event={"ID":"f6cc3890-cacb-4678-9ee5-c5611e5db8f4","Type":"ContainerDied","Data":"bc6e828f56b7826b0909b984356f1381052cde44c4969adc5db59cd8573924b4"} Jul 06 21:27:16.766053 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:16.765958 2577 scope.go:117] "RemoveContainer" containerID="a66bbcbba319eaf4a05026ff73294cf2d6eb519cd4068ecdd5a01b6a1212269b" Jul 06 21:27:16.775459 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:16.775433 2577 scope.go:117] "RemoveContainer" containerID="a66bbcbba319eaf4a05026ff73294cf2d6eb519cd4068ecdd5a01b6a1212269b" Jul 06 21:27:16.775793 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:27:16.775774 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a66bbcbba319eaf4a05026ff73294cf2d6eb519cd4068ecdd5a01b6a1212269b\": container with ID starting with a66bbcbba319eaf4a05026ff73294cf2d6eb519cd4068ecdd5a01b6a1212269b not found: ID does not exist" containerID="a66bbcbba319eaf4a05026ff73294cf2d6eb519cd4068ecdd5a01b6a1212269b" Jul 06 21:27:16.775843 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:16.775808 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a66bbcbba319eaf4a05026ff73294cf2d6eb519cd4068ecdd5a01b6a1212269b"} err="failed to get container status \"a66bbcbba319eaf4a05026ff73294cf2d6eb519cd4068ecdd5a01b6a1212269b\": rpc error: code = NotFound desc = could not find container \"a66bbcbba319eaf4a05026ff73294cf2d6eb519cd4068ecdd5a01b6a1212269b\": container with ID starting with a66bbcbba319eaf4a05026ff73294cf2d6eb519cd4068ecdd5a01b6a1212269b not found: ID does not exist" Jul 06 21:27:16.793338 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:16.793299 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-ef8479e9-59f87c5584-264d5"] Jul 06 21:27:16.799044 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:16.798998 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-ef8479e9-59f87c5584-264d5"] Jul 06 21:27:17.772294 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:17.772258 2577 generic.go:356] "Generic (PLEG): container finished" podID="e58b06de-c3ee-4284-80ee-9e5c11a89a15" containerID="099effe51bbf4476d0884a75f682e8119ab70fcc67dc9ad1c87b9dc5749c3e6b" exitCode=0 Jul 06 21:27:17.772861 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:17.772338 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" event={"ID":"e58b06de-c3ee-4284-80ee-9e5c11a89a15","Type":"ContainerDied","Data":"099effe51bbf4476d0884a75f682e8119ab70fcc67dc9ad1c87b9dc5749c3e6b"} Jul 06 21:27:17.772936 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:17.772889 2577 scope.go:117] "RemoveContainer" containerID="099effe51bbf4476d0884a75f682e8119ab70fcc67dc9ad1c87b9dc5749c3e6b" Jul 06 21:27:17.773967 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:17.773937 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:27:17.774072 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:17.774036 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:27:17.774116 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:17.774095 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:27:18.169780 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:18.169741 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6cc3890-cacb-4678-9ee5-c5611e5db8f4" path="/var/lib/kubelet/pods/f6cc3890-cacb-4678-9ee5-c5611e5db8f4/volumes" Jul 06 21:27:18.678117 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:18.678056 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:18.780356 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:18.780312 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" event={"ID":"e58b06de-c3ee-4284-80ee-9e5c11a89a15","Type":"ContainerStarted","Data":"d3e26e4ccd22bcf0b343162bd24a19db7a1c0a39b4165af6ec9871f3ab883025"} Jul 06 21:27:19.130111 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:19.130065 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:19.131576 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:19.131536 2577 patch_prober.go:28] interesting pod/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 06 21:27:19.131747 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:19.131611 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" podUID="e58b06de-c3ee-4284-80ee-9e5c11a89a15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 06 21:27:20.130132 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:20.130092 2577 patch_prober.go:28] interesting pod/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 06 21:27:20.130780 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:20.130743 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" podUID="e58b06de-c3ee-4284-80ee-9e5c11a89a15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 06 21:27:21.130141 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:21.130090 2577 patch_prober.go:28] interesting pod/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 06 21:27:21.130698 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:21.130161 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" podUID="e58b06de-c3ee-4284-80ee-9e5c11a89a15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 06 21:27:22.129585 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:22.129546 2577 patch_prober.go:28] interesting pod/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 06 21:27:22.130361 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:22.129882 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" podUID="e58b06de-c3ee-4284-80ee-9e5c11a89a15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 06 21:27:22.575544 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:22.575453 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp"] Jul 06 21:27:22.575831 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:22.575800 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" podUID="e58b06de-c3ee-4284-80ee-9e5c11a89a15" containerName="istio-proxy" containerID="cri-o://d3e26e4ccd22bcf0b343162bd24a19db7a1c0a39b4165af6ec9871f3ab883025" gracePeriod=30 Jul 06 21:27:28.231937 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.231894 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:28.355449 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.355344 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58b06de-c3ee-4284-80ee-9e5c11a89a15-workload-socket\") pod \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " Jul 06 21:27:28.355449 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.355398 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58b06de-c3ee-4284-80ee-9e5c11a89a15-istio-podinfo\") pod \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " Jul 06 21:27:28.355449 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.355413 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58b06de-c3ee-4284-80ee-9e5c11a89a15-istio-data\") pod \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " Jul 06 21:27:28.355449 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.355431 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58b06de-c3ee-4284-80ee-9e5c11a89a15-istio-envoy\") pod \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " Jul 06 21:27:28.355449 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.355450 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58b06de-c3ee-4284-80ee-9e5c11a89a15-credential-socket\") pod \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " Jul 06 21:27:28.355842 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.355562 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58b06de-c3ee-4284-80ee-9e5c11a89a15-istiod-ca-cert\") pod \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " Jul 06 21:27:28.355842 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.355606 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbpg6\" (UniqueName: \"kubernetes.io/projected/e58b06de-c3ee-4284-80ee-9e5c11a89a15-kube-api-access-jbpg6\") pod \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " Jul 06 21:27:28.355842 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.355652 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58b06de-c3ee-4284-80ee-9e5c11a89a15-workload-certs\") pod \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " Jul 06 21:27:28.355842 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.355676 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58b06de-c3ee-4284-80ee-9e5c11a89a15-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e58b06de-c3ee-4284-80ee-9e5c11a89a15" (UID: "e58b06de-c3ee-4284-80ee-9e5c11a89a15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:27:28.355842 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.355705 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58b06de-c3ee-4284-80ee-9e5c11a89a15-istio-token\") pod \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\" (UID: \"e58b06de-c3ee-4284-80ee-9e5c11a89a15\") " Jul 06 21:27:28.356142 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.355915 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58b06de-c3ee-4284-80ee-9e5c11a89a15-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e58b06de-c3ee-4284-80ee-9e5c11a89a15" (UID: "e58b06de-c3ee-4284-80ee-9e5c11a89a15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:27:28.356142 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.355974 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e58b06de-c3ee-4284-80ee-9e5c11a89a15-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e58b06de-c3ee-4284-80ee-9e5c11a89a15" (UID: "e58b06de-c3ee-4284-80ee-9e5c11a89a15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:27:28.356142 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.356115 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58b06de-c3ee-4284-80ee-9e5c11a89a15-workload-certs\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:27:28.356142 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.356117 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58b06de-c3ee-4284-80ee-9e5c11a89a15-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e58b06de-c3ee-4284-80ee-9e5c11a89a15" (UID: "e58b06de-c3ee-4284-80ee-9e5c11a89a15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:27:28.356142 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.356134 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58b06de-c3ee-4284-80ee-9e5c11a89a15-workload-socket\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:27:28.356418 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.356385 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58b06de-c3ee-4284-80ee-9e5c11a89a15-istio-data" (OuterVolumeSpecName: "istio-data") pod "e58b06de-c3ee-4284-80ee-9e5c11a89a15" (UID: "e58b06de-c3ee-4284-80ee-9e5c11a89a15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:27:28.357783 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.357758 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e58b06de-c3ee-4284-80ee-9e5c11a89a15-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e58b06de-c3ee-4284-80ee-9e5c11a89a15" (UID: "e58b06de-c3ee-4284-80ee-9e5c11a89a15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 21:27:28.357918 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.357868 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e58b06de-c3ee-4284-80ee-9e5c11a89a15-istio-token" (OuterVolumeSpecName: "istio-token") pod "e58b06de-c3ee-4284-80ee-9e5c11a89a15" (UID: "e58b06de-c3ee-4284-80ee-9e5c11a89a15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:27:28.357974 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.357950 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e58b06de-c3ee-4284-80ee-9e5c11a89a15-kube-api-access-jbpg6" (OuterVolumeSpecName: "kube-api-access-jbpg6") pod "e58b06de-c3ee-4284-80ee-9e5c11a89a15" (UID: "e58b06de-c3ee-4284-80ee-9e5c11a89a15"). InnerVolumeSpecName "kube-api-access-jbpg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:27:28.358032 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.357958 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58b06de-c3ee-4284-80ee-9e5c11a89a15-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e58b06de-c3ee-4284-80ee-9e5c11a89a15" (UID: "e58b06de-c3ee-4284-80ee-9e5c11a89a15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:27:28.457240 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.457200 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58b06de-c3ee-4284-80ee-9e5c11a89a15-istio-token\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:27:28.457240 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.457233 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58b06de-c3ee-4284-80ee-9e5c11a89a15-istio-podinfo\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:27:28.457240 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.457242 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58b06de-c3ee-4284-80ee-9e5c11a89a15-istio-data\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:27:28.457240 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.457250 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58b06de-c3ee-4284-80ee-9e5c11a89a15-istio-envoy\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:27:28.457510 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.457259 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58b06de-c3ee-4284-80ee-9e5c11a89a15-credential-socket\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:27:28.457510 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.457270 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58b06de-c3ee-4284-80ee-9e5c11a89a15-istiod-ca-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:27:28.457510 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.457285 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbpg6\" (UniqueName: \"kubernetes.io/projected/e58b06de-c3ee-4284-80ee-9e5c11a89a15-kube-api-access-jbpg6\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:27:28.835235 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.835193 2577 generic.go:356] "Generic (PLEG): container finished" podID="e58b06de-c3ee-4284-80ee-9e5c11a89a15" containerID="d3e26e4ccd22bcf0b343162bd24a19db7a1c0a39b4165af6ec9871f3ab883025" exitCode=0 Jul 06 21:27:28.835413 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.835240 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" Jul 06 21:27:28.835413 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.835274 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" event={"ID":"e58b06de-c3ee-4284-80ee-9e5c11a89a15","Type":"ContainerDied","Data":"d3e26e4ccd22bcf0b343162bd24a19db7a1c0a39b4165af6ec9871f3ab883025"} Jul 06 21:27:28.835413 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.835337 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp" event={"ID":"e58b06de-c3ee-4284-80ee-9e5c11a89a15","Type":"ContainerDied","Data":"1da9373e070a4de295eccfdb9dce6cfd514361968273011bef1d9b5d85106870"} Jul 06 21:27:28.835413 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.835367 2577 scope.go:117] "RemoveContainer" containerID="d3e26e4ccd22bcf0b343162bd24a19db7a1c0a39b4165af6ec9871f3ab883025" Jul 06 21:27:28.845485 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.845458 2577 scope.go:117] "RemoveContainer" containerID="099effe51bbf4476d0884a75f682e8119ab70fcc67dc9ad1c87b9dc5749c3e6b" Jul 06 21:27:28.854612 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.854589 2577 scope.go:117] "RemoveContainer" containerID="d3e26e4ccd22bcf0b343162bd24a19db7a1c0a39b4165af6ec9871f3ab883025" Jul 06 21:27:28.854927 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:27:28.854902 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3e26e4ccd22bcf0b343162bd24a19db7a1c0a39b4165af6ec9871f3ab883025\": container with ID starting with d3e26e4ccd22bcf0b343162bd24a19db7a1c0a39b4165af6ec9871f3ab883025 not found: ID does not exist" containerID="d3e26e4ccd22bcf0b343162bd24a19db7a1c0a39b4165af6ec9871f3ab883025" Jul 06 21:27:28.854979 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.854937 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3e26e4ccd22bcf0b343162bd24a19db7a1c0a39b4165af6ec9871f3ab883025"} err="failed to get container status \"d3e26e4ccd22bcf0b343162bd24a19db7a1c0a39b4165af6ec9871f3ab883025\": rpc error: code = NotFound desc = could not find container \"d3e26e4ccd22bcf0b343162bd24a19db7a1c0a39b4165af6ec9871f3ab883025\": container with ID starting with d3e26e4ccd22bcf0b343162bd24a19db7a1c0a39b4165af6ec9871f3ab883025 not found: ID does not exist" Jul 06 21:27:28.854979 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.854958 2577 scope.go:117] "RemoveContainer" containerID="099effe51bbf4476d0884a75f682e8119ab70fcc67dc9ad1c87b9dc5749c3e6b" Jul 06 21:27:28.855216 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:27:28.855192 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"099effe51bbf4476d0884a75f682e8119ab70fcc67dc9ad1c87b9dc5749c3e6b\": container with ID starting with 099effe51bbf4476d0884a75f682e8119ab70fcc67dc9ad1c87b9dc5749c3e6b not found: ID does not exist" containerID="099effe51bbf4476d0884a75f682e8119ab70fcc67dc9ad1c87b9dc5749c3e6b" Jul 06 21:27:28.855277 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.855215 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"099effe51bbf4476d0884a75f682e8119ab70fcc67dc9ad1c87b9dc5749c3e6b"} err="failed to get container status \"099effe51bbf4476d0884a75f682e8119ab70fcc67dc9ad1c87b9dc5749c3e6b\": rpc error: code = NotFound desc = could not find container \"099effe51bbf4476d0884a75f682e8119ab70fcc67dc9ad1c87b9dc5749c3e6b\": container with ID starting with 099effe51bbf4476d0884a75f682e8119ab70fcc67dc9ad1c87b9dc5749c3e6b not found: ID does not exist" Jul 06 21:27:28.858445 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.858410 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp"] Jul 06 21:27:28.863334 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:28.863299 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ef8479e9-openshift-default-5f5d56f8ff-gpdrp"] Jul 06 21:27:30.168872 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:30.168839 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e58b06de-c3ee-4284-80ee-9e5c11a89a15" path="/var/lib/kubelet/pods/e58b06de-c3ee-4284-80ee-9e5c11a89a15/volumes" Jul 06 21:27:43.857490 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:43.857453 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx"] Jul 06 21:27:43.857972 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:43.857870 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e58b06de-c3ee-4284-80ee-9e5c11a89a15" containerName="istio-proxy" Jul 06 21:27:43.857972 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:43.857881 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e58b06de-c3ee-4284-80ee-9e5c11a89a15" containerName="istio-proxy" Jul 06 21:27:43.857972 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:43.857897 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6cc3890-cacb-4678-9ee5-c5611e5db8f4" containerName="maas-api" Jul 06 21:27:43.857972 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:43.857902 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6cc3890-cacb-4678-9ee5-c5611e5db8f4" containerName="maas-api" Jul 06 21:27:43.857972 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:43.857911 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e58b06de-c3ee-4284-80ee-9e5c11a89a15" containerName="istio-proxy" Jul 06 21:27:43.857972 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:43.857917 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e58b06de-c3ee-4284-80ee-9e5c11a89a15" containerName="istio-proxy" Jul 06 21:27:43.860660 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:43.858583 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e58b06de-c3ee-4284-80ee-9e5c11a89a15" containerName="istio-proxy" Jul 06 21:27:43.860660 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:43.858631 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6cc3890-cacb-4678-9ee5-c5611e5db8f4" containerName="maas-api" Jul 06 21:27:43.860660 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:43.858643 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e58b06de-c3ee-4284-80ee-9e5c11a89a15" containerName="istio-proxy" Jul 06 21:27:43.863083 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:43.862387 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:43.867638 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:43.866982 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-746edc43-openshift-default-dockercfg-qf4jd\"" Jul 06 21:27:43.882069 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:43.882028 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx"] Jul 06 21:27:43.900145 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:43.900098 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c3d249-fa72-461a-a07b-e42e0d9efb53-istio-data\") pod \"e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:43.900317 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:43.900158 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c3d249-fa72-461a-a07b-e42e0d9efb53-istiod-ca-cert\") pod \"e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:43.900317 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:43.900211 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c3d249-fa72-461a-a07b-e42e0d9efb53-istio-token\") pod \"e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:43.900317 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:43.900233 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c3d249-fa72-461a-a07b-e42e0d9efb53-workload-socket\") pod \"e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:43.900317 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:43.900277 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c3d249-fa72-461a-a07b-e42e0d9efb53-istio-envoy\") pod \"e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:43.900317 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:43.900306 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c3d249-fa72-461a-a07b-e42e0d9efb53-credential-socket\") pod \"e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:43.900519 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:43.900329 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bq62\" (UniqueName: \"kubernetes.io/projected/d5c3d249-fa72-461a-a07b-e42e0d9efb53-kube-api-access-2bq62\") pod \"e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:43.900519 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:43.900368 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c3d249-fa72-461a-a07b-e42e0d9efb53-istio-podinfo\") pod \"e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:43.900519 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:43.900411 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c3d249-fa72-461a-a07b-e42e0d9efb53-workload-certs\") pod \"e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:44.001266 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:44.001211 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c3d249-fa72-461a-a07b-e42e0d9efb53-workload-certs\") pod \"e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:44.001266 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:44.001277 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c3d249-fa72-461a-a07b-e42e0d9efb53-istio-data\") pod \"e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:44.001562 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:44.001303 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c3d249-fa72-461a-a07b-e42e0d9efb53-istiod-ca-cert\") pod \"e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:44.001562 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:44.001354 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c3d249-fa72-461a-a07b-e42e0d9efb53-istio-token\") pod \"e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:44.001562 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:44.001372 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c3d249-fa72-461a-a07b-e42e0d9efb53-workload-socket\") pod \"e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:44.001562 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:44.001420 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c3d249-fa72-461a-a07b-e42e0d9efb53-istio-envoy\") pod \"e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:44.001562 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:44.001452 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c3d249-fa72-461a-a07b-e42e0d9efb53-credential-socket\") pod \"e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:44.001562 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:44.001475 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bq62\" (UniqueName: \"kubernetes.io/projected/d5c3d249-fa72-461a-a07b-e42e0d9efb53-kube-api-access-2bq62\") pod \"e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:44.001562 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:44.001515 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c3d249-fa72-461a-a07b-e42e0d9efb53-istio-podinfo\") pod \"e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:44.001947 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:44.001700 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c3d249-fa72-461a-a07b-e42e0d9efb53-workload-certs\") pod \"e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:44.001947 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:44.001772 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c3d249-fa72-461a-a07b-e42e0d9efb53-istio-data\") pod \"e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:44.002069 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:44.001957 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c3d249-fa72-461a-a07b-e42e0d9efb53-credential-socket\") pod \"e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:44.002127 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:44.002104 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c3d249-fa72-461a-a07b-e42e0d9efb53-istiod-ca-cert\") pod \"e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:44.002216 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:44.002193 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c3d249-fa72-461a-a07b-e42e0d9efb53-workload-socket\") pod \"e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:44.003878 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:44.003853 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c3d249-fa72-461a-a07b-e42e0d9efb53-istio-envoy\") pod \"e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:44.004056 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:44.004040 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c3d249-fa72-461a-a07b-e42e0d9efb53-istio-podinfo\") pod \"e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:44.016963 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:44.016930 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c3d249-fa72-461a-a07b-e42e0d9efb53-istio-token\") pod \"e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:44.017135 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:44.017041 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bq62\" (UniqueName: \"kubernetes.io/projected/d5c3d249-fa72-461a-a07b-e42e0d9efb53-kube-api-access-2bq62\") pod \"e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:44.176513 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:44.176470 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:44.336867 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:44.336835 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx"] Jul 06 21:27:44.339921 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:27:44.339895 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c3d249_fa72_461a_a07b_e42e0d9efb53.slice/crio-e7e16c9b2f66342088f2da345f5f510a8627f4a13aee220e41113dd37f3b79c5 WatchSource:0}: Error finding container e7e16c9b2f66342088f2da345f5f510a8627f4a13aee220e41113dd37f3b79c5: Status 404 returned error can't find the container with id e7e16c9b2f66342088f2da345f5f510a8627f4a13aee220e41113dd37f3b79c5 Jul 06 21:27:44.342071 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:44.342039 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:27:44.342152 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:44.342109 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:27:44.342152 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:44.342138 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:27:44.921849 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:44.921812 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" event={"ID":"d5c3d249-fa72-461a-a07b-e42e0d9efb53","Type":"ContainerStarted","Data":"f2649e6bcbad8619b5820b4fbe7c1154b9c7fdb4c690f20e484026f15d34c3d8"} Jul 06 21:27:44.922247 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:44.921865 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" event={"ID":"d5c3d249-fa72-461a-a07b-e42e0d9efb53","Type":"ContainerStarted","Data":"e7e16c9b2f66342088f2da345f5f510a8627f4a13aee220e41113dd37f3b79c5"} Jul 06 21:27:44.949244 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:44.949190 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" podStartSLOduration=1.949163873 podStartE2EDuration="1.949163873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:27:44.945984934 +0000 UTC m=+1749.264208193" watchObservedRunningTime="2026-07-06 21:27:44.949163873 +0000 UTC m=+1749.267387132" Jul 06 21:27:45.177140 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:45.177035 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:45.178680 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:45.178652 2577 patch_prober.go:28] interesting pod/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 06 21:27:45.178803 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:45.178705 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" podUID="d5c3d249-fa72-461a-a07b-e42e0d9efb53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 06 21:27:46.181638 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:46.181600 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:46.932739 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:46.932703 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:46.933640 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:46.933615 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:27:47.132594 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:47.132548 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-746edc43-fb884f549-rvrfl"] Jul 06 21:27:47.135931 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:47.135906 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-746edc43-fb884f549-rvrfl" Jul 06 21:27:47.141048 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:47.140995 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-746edc43-serving-cert\"" Jul 06 21:27:47.162823 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:47.162787 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-746edc43-fb884f549-rvrfl"] Jul 06 21:27:47.233875 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:47.233767 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97q9x\" (UniqueName: \"kubernetes.io/projected/1847ee0f-462a-4ebe-8f7a-4c693ae1554f-kube-api-access-97q9x\") pod \"maas-api-e2e-mt-746edc43-fb884f549-rvrfl\" (UID: \"1847ee0f-462a-4ebe-8f7a-4c693ae1554f\") " pod="opendatahub/maas-api-e2e-mt-746edc43-fb884f549-rvrfl" Jul 06 21:27:47.233875 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:47.233809 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1847ee0f-462a-4ebe-8f7a-4c693ae1554f-maas-api-tls\") pod \"maas-api-e2e-mt-746edc43-fb884f549-rvrfl\" (UID: \"1847ee0f-462a-4ebe-8f7a-4c693ae1554f\") " pod="opendatahub/maas-api-e2e-mt-746edc43-fb884f549-rvrfl" Jul 06 21:27:47.234325 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:47.233955 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1847ee0f-462a-4ebe-8f7a-4c693ae1554f-openshift-service-ca\") pod \"maas-api-e2e-mt-746edc43-fb884f549-rvrfl\" (UID: \"1847ee0f-462a-4ebe-8f7a-4c693ae1554f\") " pod="opendatahub/maas-api-e2e-mt-746edc43-fb884f549-rvrfl" Jul 06 21:27:47.335196 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:47.335151 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1847ee0f-462a-4ebe-8f7a-4c693ae1554f-openshift-service-ca\") pod \"maas-api-e2e-mt-746edc43-fb884f549-rvrfl\" (UID: \"1847ee0f-462a-4ebe-8f7a-4c693ae1554f\") " pod="opendatahub/maas-api-e2e-mt-746edc43-fb884f549-rvrfl" Jul 06 21:27:47.335359 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:47.335224 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97q9x\" (UniqueName: \"kubernetes.io/projected/1847ee0f-462a-4ebe-8f7a-4c693ae1554f-kube-api-access-97q9x\") pod \"maas-api-e2e-mt-746edc43-fb884f549-rvrfl\" (UID: \"1847ee0f-462a-4ebe-8f7a-4c693ae1554f\") " pod="opendatahub/maas-api-e2e-mt-746edc43-fb884f549-rvrfl" Jul 06 21:27:47.335359 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:47.335251 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1847ee0f-462a-4ebe-8f7a-4c693ae1554f-maas-api-tls\") pod \"maas-api-e2e-mt-746edc43-fb884f549-rvrfl\" (UID: \"1847ee0f-462a-4ebe-8f7a-4c693ae1554f\") " pod="opendatahub/maas-api-e2e-mt-746edc43-fb884f549-rvrfl" Jul 06 21:27:47.335859 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:47.335835 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1847ee0f-462a-4ebe-8f7a-4c693ae1554f-openshift-service-ca\") pod \"maas-api-e2e-mt-746edc43-fb884f549-rvrfl\" (UID: \"1847ee0f-462a-4ebe-8f7a-4c693ae1554f\") " pod="opendatahub/maas-api-e2e-mt-746edc43-fb884f549-rvrfl" Jul 06 21:27:47.337794 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:47.337772 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1847ee0f-462a-4ebe-8f7a-4c693ae1554f-maas-api-tls\") pod \"maas-api-e2e-mt-746edc43-fb884f549-rvrfl\" (UID: \"1847ee0f-462a-4ebe-8f7a-4c693ae1554f\") " pod="opendatahub/maas-api-e2e-mt-746edc43-fb884f549-rvrfl" Jul 06 21:27:47.348758 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:47.348729 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97q9x\" (UniqueName: \"kubernetes.io/projected/1847ee0f-462a-4ebe-8f7a-4c693ae1554f-kube-api-access-97q9x\") pod \"maas-api-e2e-mt-746edc43-fb884f549-rvrfl\" (UID: \"1847ee0f-462a-4ebe-8f7a-4c693ae1554f\") " pod="opendatahub/maas-api-e2e-mt-746edc43-fb884f549-rvrfl" Jul 06 21:27:47.451263 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:47.451227 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-746edc43-fb884f549-rvrfl" Jul 06 21:27:47.610778 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:47.610664 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-746edc43-fb884f549-rvrfl"] Jul 06 21:27:47.612939 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:27:47.612909 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1847ee0f_462a_4ebe_8f7a_4c693ae1554f.slice/crio-2066cc3b671d554b77f2341cb720573637777eb157ca3ecf40006ecb9c1dc06f WatchSource:0}: Error finding container 2066cc3b671d554b77f2341cb720573637777eb157ca3ecf40006ecb9c1dc06f: Status 404 returned error can't find the container with id 2066cc3b671d554b77f2341cb720573637777eb157ca3ecf40006ecb9c1dc06f Jul 06 21:27:47.939775 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:47.939730 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-746edc43-fb884f549-rvrfl" event={"ID":"1847ee0f-462a-4ebe-8f7a-4c693ae1554f","Type":"ContainerStarted","Data":"2066cc3b671d554b77f2341cb720573637777eb157ca3ecf40006ecb9c1dc06f"} Jul 06 21:27:48.947121 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:48.947075 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-746edc43-fb884f549-rvrfl" event={"ID":"1847ee0f-462a-4ebe-8f7a-4c693ae1554f","Type":"ContainerStarted","Data":"144a58892cde2697f7148027334335bda55d0a6cd28de4355d049b18d53b2937"} Jul 06 21:27:48.967304 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:48.967244 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-746edc43-fb884f549-rvrfl" podStartSLOduration=1.6534829370000002 podStartE2EDuration="1.967224269s" totalImagesPullingTime="313.741332ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:27:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:27:48.964842244 +0000 UTC m=+1753.283065517" watchObservedRunningTime="2026-07-06 21:27:48.967224269 +0000 UTC m=+1753.285447580" Jul 06 21:27:49.952913 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:49.952881 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-746edc43-fb884f549-rvrfl" Jul 06 21:27:55.964574 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:27:55.964541 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-746edc43-fb884f549-rvrfl" Jul 06 21:28:00.013824 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:00.013790 2577 generic.go:356] "Generic (PLEG): container finished" podID="d5c3d249-fa72-461a-a07b-e42e0d9efb53" containerID="f2649e6bcbad8619b5820b4fbe7c1154b9c7fdb4c690f20e484026f15d34c3d8" exitCode=0 Jul 06 21:28:00.014430 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:00.013864 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" event={"ID":"d5c3d249-fa72-461a-a07b-e42e0d9efb53","Type":"ContainerDied","Data":"f2649e6bcbad8619b5820b4fbe7c1154b9c7fdb4c690f20e484026f15d34c3d8"} Jul 06 21:28:00.014513 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:00.014494 2577 scope.go:117] "RemoveContainer" containerID="f2649e6bcbad8619b5820b4fbe7c1154b9c7fdb4c690f20e484026f15d34c3d8" Jul 06 21:28:00.015624 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:00.015596 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:28:00.015719 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:00.015701 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:28:00.015772 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:00.015758 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:28:01.021656 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:01.021621 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" event={"ID":"d5c3d249-fa72-461a-a07b-e42e0d9efb53","Type":"ContainerStarted","Data":"ef9a41c439f320b0ddafce076d4fffcf4fe79aabd66bd3b3e6cf8eead32dcea5"} Jul 06 21:28:01.176739 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:01.176701 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:28:01.177772 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:01.177746 2577 patch_prober.go:28] interesting pod/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 06 21:28:01.177887 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:01.177797 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" podUID="d5c3d249-fa72-461a-a07b-e42e0d9efb53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 06 21:28:01.944328 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:01.944237 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:28:02.177413 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:02.177378 2577 patch_prober.go:28] interesting pod/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 06 21:28:02.177854 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:02.177432 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" podUID="d5c3d249-fa72-461a-a07b-e42e0d9efb53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 06 21:28:03.177819 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:03.177783 2577 patch_prober.go:28] interesting pod/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 06 21:28:03.178261 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:03.177841 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" podUID="d5c3d249-fa72-461a-a07b-e42e0d9efb53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 06 21:28:04.177361 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:04.177318 2577 patch_prober.go:28] interesting pod/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 06 21:28:04.177553 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:04.177383 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" podUID="d5c3d249-fa72-461a-a07b-e42e0d9efb53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 06 21:28:05.199250 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:05.199200 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:28:05.200547 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:05.200519 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:28:25.585214 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:25.585173 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-746edc43-fb884f549-rvrfl"] Jul 06 21:28:25.585760 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:25.585376 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-746edc43-fb884f549-rvrfl" podUID="1847ee0f-462a-4ebe-8f7a-4c693ae1554f" containerName="maas-api" containerID="cri-o://144a58892cde2697f7148027334335bda55d0a6cd28de4355d049b18d53b2937" gracePeriod=30 Jul 06 21:28:25.839858 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:25.839786 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-746edc43-fb884f549-rvrfl" Jul 06 21:28:25.911207 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:25.911169 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97q9x\" (UniqueName: \"kubernetes.io/projected/1847ee0f-462a-4ebe-8f7a-4c693ae1554f-kube-api-access-97q9x\") pod \"1847ee0f-462a-4ebe-8f7a-4c693ae1554f\" (UID: \"1847ee0f-462a-4ebe-8f7a-4c693ae1554f\") " Jul 06 21:28:25.911207 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:25.911207 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1847ee0f-462a-4ebe-8f7a-4c693ae1554f-maas-api-tls\") pod \"1847ee0f-462a-4ebe-8f7a-4c693ae1554f\" (UID: \"1847ee0f-462a-4ebe-8f7a-4c693ae1554f\") " Jul 06 21:28:25.911470 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:25.911277 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1847ee0f-462a-4ebe-8f7a-4c693ae1554f-openshift-service-ca\") pod \"1847ee0f-462a-4ebe-8f7a-4c693ae1554f\" (UID: \"1847ee0f-462a-4ebe-8f7a-4c693ae1554f\") " Jul 06 21:28:25.911686 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:25.911649 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1847ee0f-462a-4ebe-8f7a-4c693ae1554f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1847ee0f-462a-4ebe-8f7a-4c693ae1554f" (UID: "1847ee0f-462a-4ebe-8f7a-4c693ae1554f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:28:25.913332 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:25.913307 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1847ee0f-462a-4ebe-8f7a-4c693ae1554f-kube-api-access-97q9x" (OuterVolumeSpecName: "kube-api-access-97q9x") pod "1847ee0f-462a-4ebe-8f7a-4c693ae1554f" (UID: "1847ee0f-462a-4ebe-8f7a-4c693ae1554f"). InnerVolumeSpecName "kube-api-access-97q9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:28:25.913441 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:25.913312 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1847ee0f-462a-4ebe-8f7a-4c693ae1554f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1847ee0f-462a-4ebe-8f7a-4c693ae1554f" (UID: "1847ee0f-462a-4ebe-8f7a-4c693ae1554f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:28:26.012201 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:26.012170 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1847ee0f-462a-4ebe-8f7a-4c693ae1554f-openshift-service-ca\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:28:26.012201 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:26.012199 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97q9x\" (UniqueName: \"kubernetes.io/projected/1847ee0f-462a-4ebe-8f7a-4c693ae1554f-kube-api-access-97q9x\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:28:26.012201 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:26.012212 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1847ee0f-462a-4ebe-8f7a-4c693ae1554f-maas-api-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:28:26.170469 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:26.170427 2577 generic.go:356] "Generic (PLEG): container finished" podID="1847ee0f-462a-4ebe-8f7a-4c693ae1554f" containerID="144a58892cde2697f7148027334335bda55d0a6cd28de4355d049b18d53b2937" exitCode=0 Jul 06 21:28:26.170469 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:26.170465 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-746edc43-fb884f549-rvrfl" Jul 06 21:28:26.170698 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:26.170504 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-746edc43-fb884f549-rvrfl" event={"ID":"1847ee0f-462a-4ebe-8f7a-4c693ae1554f","Type":"ContainerDied","Data":"144a58892cde2697f7148027334335bda55d0a6cd28de4355d049b18d53b2937"} Jul 06 21:28:26.170698 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:26.170555 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-746edc43-fb884f549-rvrfl" event={"ID":"1847ee0f-462a-4ebe-8f7a-4c693ae1554f","Type":"ContainerDied","Data":"2066cc3b671d554b77f2341cb720573637777eb157ca3ecf40006ecb9c1dc06f"} Jul 06 21:28:26.170698 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:26.170578 2577 scope.go:117] "RemoveContainer" containerID="144a58892cde2697f7148027334335bda55d0a6cd28de4355d049b18d53b2937" Jul 06 21:28:26.180394 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:26.180367 2577 scope.go:117] "RemoveContainer" containerID="144a58892cde2697f7148027334335bda55d0a6cd28de4355d049b18d53b2937" Jul 06 21:28:26.180748 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:28:26.180725 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"144a58892cde2697f7148027334335bda55d0a6cd28de4355d049b18d53b2937\": container with ID starting with 144a58892cde2697f7148027334335bda55d0a6cd28de4355d049b18d53b2937 not found: ID does not exist" containerID="144a58892cde2697f7148027334335bda55d0a6cd28de4355d049b18d53b2937" Jul 06 21:28:26.180846 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:26.180759 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"144a58892cde2697f7148027334335bda55d0a6cd28de4355d049b18d53b2937"} err="failed to get container status \"144a58892cde2697f7148027334335bda55d0a6cd28de4355d049b18d53b2937\": rpc error: code = NotFound desc = could not find container \"144a58892cde2697f7148027334335bda55d0a6cd28de4355d049b18d53b2937\": container with ID starting with 144a58892cde2697f7148027334335bda55d0a6cd28de4355d049b18d53b2937 not found: ID does not exist" Jul 06 21:28:26.203076 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:26.203027 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-746edc43-fb884f549-rvrfl"] Jul 06 21:28:26.210058 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:26.210021 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-746edc43-fb884f549-rvrfl"] Jul 06 21:28:28.168869 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:28.168822 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1847ee0f-462a-4ebe-8f7a-4c693ae1554f" path="/var/lib/kubelet/pods/1847ee0f-462a-4ebe-8f7a-4c693ae1554f/volumes" Jul 06 21:28:36.295778 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:36.295740 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qjs2n_64b45669-ed56-41b9-a3ed-1fe5c13508b6/ovn-acl-logging/0.log" Jul 06 21:28:36.298119 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:28:36.298095 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qjs2n_64b45669-ed56-41b9-a3ed-1fe5c13508b6/ovn-acl-logging/0.log" Jul 06 21:29:35.513136 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:35.513097 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx"] Jul 06 21:29:35.513689 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:35.513377 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" podUID="d5c3d249-fa72-461a-a07b-e42e0d9efb53" containerName="istio-proxy" containerID="cri-o://ef9a41c439f320b0ddafce076d4fffcf4fe79aabd66bd3b3e6cf8eead32dcea5" gracePeriod=30 Jul 06 21:29:40.778383 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:40.778301 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:29:40.947276 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:40.943875 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c3d249-fa72-461a-a07b-e42e0d9efb53-credential-socket\") pod \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " Jul 06 21:29:40.947276 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:40.943939 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c3d249-fa72-461a-a07b-e42e0d9efb53-istio-podinfo\") pod \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " Jul 06 21:29:40.947276 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:40.943996 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c3d249-fa72-461a-a07b-e42e0d9efb53-istiod-ca-cert\") pod \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " Jul 06 21:29:40.947276 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:40.944065 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c3d249-fa72-461a-a07b-e42e0d9efb53-workload-certs\") pod \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " Jul 06 21:29:40.947276 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:40.944099 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c3d249-fa72-461a-a07b-e42e0d9efb53-istio-token\") pod \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " Jul 06 21:29:40.947276 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:40.944135 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c3d249-fa72-461a-a07b-e42e0d9efb53-workload-socket\") pod \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " Jul 06 21:29:40.947276 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:40.944183 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c3d249-fa72-461a-a07b-e42e0d9efb53-istio-envoy\") pod \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " Jul 06 21:29:40.947276 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:40.944244 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c3d249-fa72-461a-a07b-e42e0d9efb53-istio-data\") pod \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " Jul 06 21:29:40.947276 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:40.944356 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bq62\" (UniqueName: \"kubernetes.io/projected/d5c3d249-fa72-461a-a07b-e42e0d9efb53-kube-api-access-2bq62\") pod \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\" (UID: \"d5c3d249-fa72-461a-a07b-e42e0d9efb53\") " Jul 06 21:29:40.948930 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:40.948346 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c3d249-fa72-461a-a07b-e42e0d9efb53-kube-api-access-2bq62" (OuterVolumeSpecName: "kube-api-access-2bq62") pod "d5c3d249-fa72-461a-a07b-e42e0d9efb53" (UID: "d5c3d249-fa72-461a-a07b-e42e0d9efb53"). InnerVolumeSpecName "kube-api-access-2bq62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:29:40.948930 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:40.948635 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c3d249-fa72-461a-a07b-e42e0d9efb53-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d5c3d249-fa72-461a-a07b-e42e0d9efb53" (UID: "d5c3d249-fa72-461a-a07b-e42e0d9efb53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:29:40.951094 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:40.951038 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c3d249-fa72-461a-a07b-e42e0d9efb53-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d5c3d249-fa72-461a-a07b-e42e0d9efb53" (UID: "d5c3d249-fa72-461a-a07b-e42e0d9efb53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:29:40.951257 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:40.951126 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c3d249-fa72-461a-a07b-e42e0d9efb53-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d5c3d249-fa72-461a-a07b-e42e0d9efb53" (UID: "d5c3d249-fa72-461a-a07b-e42e0d9efb53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:29:40.951338 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:40.951167 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c3d249-fa72-461a-a07b-e42e0d9efb53-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d5c3d249-fa72-461a-a07b-e42e0d9efb53" (UID: "d5c3d249-fa72-461a-a07b-e42e0d9efb53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:29:40.951448 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:40.951429 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5c3d249-fa72-461a-a07b-e42e0d9efb53-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d5c3d249-fa72-461a-a07b-e42e0d9efb53" (UID: "d5c3d249-fa72-461a-a07b-e42e0d9efb53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 21:29:40.952054 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:40.951994 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c3d249-fa72-461a-a07b-e42e0d9efb53-istio-data" (OuterVolumeSpecName: "istio-data") pod "d5c3d249-fa72-461a-a07b-e42e0d9efb53" (UID: "d5c3d249-fa72-461a-a07b-e42e0d9efb53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:29:40.952730 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:40.952705 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c3d249-fa72-461a-a07b-e42e0d9efb53-istio-token" (OuterVolumeSpecName: "istio-token") pod "d5c3d249-fa72-461a-a07b-e42e0d9efb53" (UID: "d5c3d249-fa72-461a-a07b-e42e0d9efb53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:29:40.952941 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:40.952921 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c3d249-fa72-461a-a07b-e42e0d9efb53-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d5c3d249-fa72-461a-a07b-e42e0d9efb53" (UID: "d5c3d249-fa72-461a-a07b-e42e0d9efb53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:29:41.046176 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:41.046079 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bq62\" (UniqueName: \"kubernetes.io/projected/d5c3d249-fa72-461a-a07b-e42e0d9efb53-kube-api-access-2bq62\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:29:41.046176 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:41.046114 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c3d249-fa72-461a-a07b-e42e0d9efb53-credential-socket\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:29:41.046176 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:41.046124 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5c3d249-fa72-461a-a07b-e42e0d9efb53-istio-podinfo\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:29:41.046176 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:41.046134 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5c3d249-fa72-461a-a07b-e42e0d9efb53-istiod-ca-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:29:41.046176 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:41.046143 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c3d249-fa72-461a-a07b-e42e0d9efb53-workload-certs\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:29:41.046176 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:41.046152 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c3d249-fa72-461a-a07b-e42e0d9efb53-istio-token\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:29:41.046176 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:41.046159 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5c3d249-fa72-461a-a07b-e42e0d9efb53-workload-socket\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:29:41.046176 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:41.046168 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5c3d249-fa72-461a-a07b-e42e0d9efb53-istio-envoy\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:29:41.046176 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:41.046175 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5c3d249-fa72-461a-a07b-e42e0d9efb53-istio-data\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:29:41.555575 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:41.555535 2577 generic.go:356] "Generic (PLEG): container finished" podID="d5c3d249-fa72-461a-a07b-e42e0d9efb53" containerID="ef9a41c439f320b0ddafce076d4fffcf4fe79aabd66bd3b3e6cf8eead32dcea5" exitCode=0 Jul 06 21:29:41.555849 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:41.555609 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" Jul 06 21:29:41.555849 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:41.555625 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" event={"ID":"d5c3d249-fa72-461a-a07b-e42e0d9efb53","Type":"ContainerDied","Data":"ef9a41c439f320b0ddafce076d4fffcf4fe79aabd66bd3b3e6cf8eead32dcea5"} Jul 06 21:29:41.555849 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:41.555689 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx" event={"ID":"d5c3d249-fa72-461a-a07b-e42e0d9efb53","Type":"ContainerDied","Data":"e7e16c9b2f66342088f2da345f5f510a8627f4a13aee220e41113dd37f3b79c5"} Jul 06 21:29:41.555849 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:41.555711 2577 scope.go:117] "RemoveContainer" containerID="ef9a41c439f320b0ddafce076d4fffcf4fe79aabd66bd3b3e6cf8eead32dcea5" Jul 06 21:29:41.565088 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:41.565028 2577 scope.go:117] "RemoveContainer" containerID="f2649e6bcbad8619b5820b4fbe7c1154b9c7fdb4c690f20e484026f15d34c3d8" Jul 06 21:29:41.573746 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:41.573723 2577 scope.go:117] "RemoveContainer" containerID="ef9a41c439f320b0ddafce076d4fffcf4fe79aabd66bd3b3e6cf8eead32dcea5" Jul 06 21:29:41.574092 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:29:41.574071 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef9a41c439f320b0ddafce076d4fffcf4fe79aabd66bd3b3e6cf8eead32dcea5\": container with ID starting with ef9a41c439f320b0ddafce076d4fffcf4fe79aabd66bd3b3e6cf8eead32dcea5 not found: ID does not exist" containerID="ef9a41c439f320b0ddafce076d4fffcf4fe79aabd66bd3b3e6cf8eead32dcea5" Jul 06 21:29:41.574176 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:41.574107 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef9a41c439f320b0ddafce076d4fffcf4fe79aabd66bd3b3e6cf8eead32dcea5"} err="failed to get container status \"ef9a41c439f320b0ddafce076d4fffcf4fe79aabd66bd3b3e6cf8eead32dcea5\": rpc error: code = NotFound desc = could not find container \"ef9a41c439f320b0ddafce076d4fffcf4fe79aabd66bd3b3e6cf8eead32dcea5\": container with ID starting with ef9a41c439f320b0ddafce076d4fffcf4fe79aabd66bd3b3e6cf8eead32dcea5 not found: ID does not exist" Jul 06 21:29:41.574176 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:41.574135 2577 scope.go:117] "RemoveContainer" containerID="f2649e6bcbad8619b5820b4fbe7c1154b9c7fdb4c690f20e484026f15d34c3d8" Jul 06 21:29:41.574424 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:29:41.574408 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2649e6bcbad8619b5820b4fbe7c1154b9c7fdb4c690f20e484026f15d34c3d8\": container with ID starting with f2649e6bcbad8619b5820b4fbe7c1154b9c7fdb4c690f20e484026f15d34c3d8 not found: ID does not exist" containerID="f2649e6bcbad8619b5820b4fbe7c1154b9c7fdb4c690f20e484026f15d34c3d8" Jul 06 21:29:41.574480 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:41.574432 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2649e6bcbad8619b5820b4fbe7c1154b9c7fdb4c690f20e484026f15d34c3d8"} err="failed to get container status \"f2649e6bcbad8619b5820b4fbe7c1154b9c7fdb4c690f20e484026f15d34c3d8\": rpc error: code = NotFound desc = could not find container \"f2649e6bcbad8619b5820b4fbe7c1154b9c7fdb4c690f20e484026f15d34c3d8\": container with ID starting with f2649e6bcbad8619b5820b4fbe7c1154b9c7fdb4c690f20e484026f15d34c3d8 not found: ID does not exist" Jul 06 21:29:41.584230 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:41.584191 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx"] Jul 06 21:29:41.591637 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:41.591600 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-746edc43-openshift-default-68ff569d47-fb9qx"] Jul 06 21:29:42.171153 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:29:42.171119 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5c3d249-fa72-461a-a07b-e42e0d9efb53" path="/var/lib/kubelet/pods/d5c3d249-fa72-461a-a07b-e42e0d9efb53/volumes" Jul 06 21:30:00.142668 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:00.142584 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722890-lmd47"] Jul 06 21:30:00.143084 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:00.142981 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1847ee0f-462a-4ebe-8f7a-4c693ae1554f" containerName="maas-api" Jul 06 21:30:00.143084 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:00.142992 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1847ee0f-462a-4ebe-8f7a-4c693ae1554f" containerName="maas-api" Jul 06 21:30:00.143084 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:00.143028 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5c3d249-fa72-461a-a07b-e42e0d9efb53" containerName="istio-proxy" Jul 06 21:30:00.143084 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:00.143038 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5c3d249-fa72-461a-a07b-e42e0d9efb53" containerName="istio-proxy" Jul 06 21:30:00.143232 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:00.143107 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5c3d249-fa72-461a-a07b-e42e0d9efb53" containerName="istio-proxy" Jul 06 21:30:00.143232 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:00.143115 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1847ee0f-462a-4ebe-8f7a-4c693ae1554f" containerName="maas-api" Jul 06 21:30:00.143232 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:00.143124 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5c3d249-fa72-461a-a07b-e42e0d9efb53" containerName="istio-proxy" Jul 06 21:30:00.147863 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:00.147840 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722890-lmd47" Jul 06 21:30:00.157821 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:00.157786 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722890-lmd47"] Jul 06 21:30:00.225355 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:00.225305 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkbwm\" (UniqueName: \"kubernetes.io/projected/303aa271-951a-4bbd-aa49-33adbf5b4d28-kube-api-access-pkbwm\") pod \"maas-api-key-cleanup-29722890-lmd47\" (UID: \"303aa271-951a-4bbd-aa49-33adbf5b4d28\") " pod="opendatahub/maas-api-key-cleanup-29722890-lmd47" Jul 06 21:30:00.325875 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:00.325834 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkbwm\" (UniqueName: \"kubernetes.io/projected/303aa271-951a-4bbd-aa49-33adbf5b4d28-kube-api-access-pkbwm\") pod \"maas-api-key-cleanup-29722890-lmd47\" (UID: \"303aa271-951a-4bbd-aa49-33adbf5b4d28\") " pod="opendatahub/maas-api-key-cleanup-29722890-lmd47" Jul 06 21:30:00.335223 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:00.335183 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkbwm\" (UniqueName: \"kubernetes.io/projected/303aa271-951a-4bbd-aa49-33adbf5b4d28-kube-api-access-pkbwm\") pod \"maas-api-key-cleanup-29722890-lmd47\" (UID: \"303aa271-951a-4bbd-aa49-33adbf5b4d28\") " pod="opendatahub/maas-api-key-cleanup-29722890-lmd47" Jul 06 21:30:00.463551 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:00.463464 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722890-lmd47" Jul 06 21:30:00.605292 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:00.605264 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722890-lmd47"] Jul 06 21:30:00.606940 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:30:00.606898 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303aa271_951a_4bbd_aa49_33adbf5b4d28.slice/crio-1ca7c1d2bd34de73006ca8d5ee543443ccda7a87d0e1e1f087358e97c23d314a WatchSource:0}: Error finding container 1ca7c1d2bd34de73006ca8d5ee543443ccda7a87d0e1e1f087358e97c23d314a: Status 404 returned error can't find the container with id 1ca7c1d2bd34de73006ca8d5ee543443ccda7a87d0e1e1f087358e97c23d314a Jul 06 21:30:00.653615 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:00.653580 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722890-lmd47" event={"ID":"303aa271-951a-4bbd-aa49-33adbf5b4d28","Type":"ContainerStarted","Data":"1ca7c1d2bd34de73006ca8d5ee543443ccda7a87d0e1e1f087358e97c23d314a"} Jul 06 21:30:01.660210 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:01.660177 2577 generic.go:356] "Generic (PLEG): container finished" podID="303aa271-951a-4bbd-aa49-33adbf5b4d28" containerID="5c00428724184d4b14d985b003df47f6e2c271dc43e3638bc9c0b8cb4da7855f" exitCode=0 Jul 06 21:30:01.660591 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:01.660244 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722890-lmd47" event={"ID":"303aa271-951a-4bbd-aa49-33adbf5b4d28","Type":"ContainerDied","Data":"5c00428724184d4b14d985b003df47f6e2c271dc43e3638bc9c0b8cb4da7855f"} Jul 06 21:30:02.805800 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:02.805765 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722890-lmd47" Jul 06 21:30:02.948681 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:02.948579 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkbwm\" (UniqueName: \"kubernetes.io/projected/303aa271-951a-4bbd-aa49-33adbf5b4d28-kube-api-access-pkbwm\") pod \"303aa271-951a-4bbd-aa49-33adbf5b4d28\" (UID: \"303aa271-951a-4bbd-aa49-33adbf5b4d28\") " Jul 06 21:30:02.950711 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:02.950676 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/303aa271-951a-4bbd-aa49-33adbf5b4d28-kube-api-access-pkbwm" (OuterVolumeSpecName: "kube-api-access-pkbwm") pod "303aa271-951a-4bbd-aa49-33adbf5b4d28" (UID: "303aa271-951a-4bbd-aa49-33adbf5b4d28"). InnerVolumeSpecName "kube-api-access-pkbwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:30:03.050077 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:03.050040 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkbwm\" (UniqueName: \"kubernetes.io/projected/303aa271-951a-4bbd-aa49-33adbf5b4d28-kube-api-access-pkbwm\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:30:03.671999 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:03.671955 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722890-lmd47" event={"ID":"303aa271-951a-4bbd-aa49-33adbf5b4d28","Type":"ContainerDied","Data":"1ca7c1d2bd34de73006ca8d5ee543443ccda7a87d0e1e1f087358e97c23d314a"} Jul 06 21:30:03.672196 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:03.672027 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ca7c1d2bd34de73006ca8d5ee543443ccda7a87d0e1e1f087358e97c23d314a" Jul 06 21:30:03.672196 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:03.671968 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722890-lmd47" Jul 06 21:30:31.524363 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.524320 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2"] Jul 06 21:30:31.525706 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.525553 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5c3d249-fa72-461a-a07b-e42e0d9efb53" containerName="istio-proxy" Jul 06 21:30:31.525706 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.525580 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5c3d249-fa72-461a-a07b-e42e0d9efb53" containerName="istio-proxy" Jul 06 21:30:31.525706 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.525606 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="303aa271-951a-4bbd-aa49-33adbf5b4d28" containerName="cleanup" Jul 06 21:30:31.525706 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.525612 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="303aa271-951a-4bbd-aa49-33adbf5b4d28" containerName="cleanup" Jul 06 21:30:31.526044 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.525735 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="303aa271-951a-4bbd-aa49-33adbf5b4d28" containerName="cleanup" Jul 06 21:30:31.529275 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.529251 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:31.531917 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.531889 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-078357-openshift-default-dockercfg-qc8cv\"" Jul 06 21:30:31.544547 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.544511 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2"] Jul 06 21:30:31.610143 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.610096 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9219d2-e2a1-49a3-9068-cc5234ac746c-workload-certs\") pod \"e2e-inf-a-078357-openshift-default-777d8b777-mjlh2\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:31.610143 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.610144 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9219d2-e2a1-49a3-9068-cc5234ac746c-istio-token\") pod \"e2e-inf-a-078357-openshift-default-777d8b777-mjlh2\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:31.610395 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.610197 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9219d2-e2a1-49a3-9068-cc5234ac746c-workload-socket\") pod \"e2e-inf-a-078357-openshift-default-777d8b777-mjlh2\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:31.610395 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.610309 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsqzd\" (UniqueName: \"kubernetes.io/projected/fd9219d2-e2a1-49a3-9068-cc5234ac746c-kube-api-access-nsqzd\") pod \"e2e-inf-a-078357-openshift-default-777d8b777-mjlh2\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:31.610395 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.610368 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9219d2-e2a1-49a3-9068-cc5234ac746c-istiod-ca-cert\") pod \"e2e-inf-a-078357-openshift-default-777d8b777-mjlh2\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:31.610496 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.610420 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9219d2-e2a1-49a3-9068-cc5234ac746c-istio-envoy\") pod \"e2e-inf-a-078357-openshift-default-777d8b777-mjlh2\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:31.610496 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.610440 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9219d2-e2a1-49a3-9068-cc5234ac746c-istio-data\") pod \"e2e-inf-a-078357-openshift-default-777d8b777-mjlh2\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:31.610582 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.610498 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9219d2-e2a1-49a3-9068-cc5234ac746c-credential-socket\") pod \"e2e-inf-a-078357-openshift-default-777d8b777-mjlh2\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:31.610582 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.610562 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9219d2-e2a1-49a3-9068-cc5234ac746c-istio-podinfo\") pod \"e2e-inf-a-078357-openshift-default-777d8b777-mjlh2\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:31.711499 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.711457 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9219d2-e2a1-49a3-9068-cc5234ac746c-istio-envoy\") pod \"e2e-inf-a-078357-openshift-default-777d8b777-mjlh2\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:31.711499 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.711497 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9219d2-e2a1-49a3-9068-cc5234ac746c-istio-data\") pod \"e2e-inf-a-078357-openshift-default-777d8b777-mjlh2\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:31.711726 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.711540 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9219d2-e2a1-49a3-9068-cc5234ac746c-credential-socket\") pod \"e2e-inf-a-078357-openshift-default-777d8b777-mjlh2\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:31.711726 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.711599 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9219d2-e2a1-49a3-9068-cc5234ac746c-istio-podinfo\") pod \"e2e-inf-a-078357-openshift-default-777d8b777-mjlh2\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:31.711726 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.711661 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9219d2-e2a1-49a3-9068-cc5234ac746c-workload-certs\") pod \"e2e-inf-a-078357-openshift-default-777d8b777-mjlh2\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:31.711726 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.711700 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9219d2-e2a1-49a3-9068-cc5234ac746c-istio-token\") pod \"e2e-inf-a-078357-openshift-default-777d8b777-mjlh2\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:31.711901 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.711755 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9219d2-e2a1-49a3-9068-cc5234ac746c-workload-socket\") pod \"e2e-inf-a-078357-openshift-default-777d8b777-mjlh2\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:31.711901 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.711826 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsqzd\" (UniqueName: \"kubernetes.io/projected/fd9219d2-e2a1-49a3-9068-cc5234ac746c-kube-api-access-nsqzd\") pod \"e2e-inf-a-078357-openshift-default-777d8b777-mjlh2\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:31.711901 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.711863 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9219d2-e2a1-49a3-9068-cc5234ac746c-istiod-ca-cert\") pod \"e2e-inf-a-078357-openshift-default-777d8b777-mjlh2\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:31.712087 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.712045 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9219d2-e2a1-49a3-9068-cc5234ac746c-istio-data\") pod \"e2e-inf-a-078357-openshift-default-777d8b777-mjlh2\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:31.712087 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.712058 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9219d2-e2a1-49a3-9068-cc5234ac746c-credential-socket\") pod \"e2e-inf-a-078357-openshift-default-777d8b777-mjlh2\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:31.712188 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.712162 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9219d2-e2a1-49a3-9068-cc5234ac746c-workload-socket\") pod \"e2e-inf-a-078357-openshift-default-777d8b777-mjlh2\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:31.712424 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.712401 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9219d2-e2a1-49a3-9068-cc5234ac746c-workload-certs\") pod \"e2e-inf-a-078357-openshift-default-777d8b777-mjlh2\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:31.712532 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.712515 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9219d2-e2a1-49a3-9068-cc5234ac746c-istiod-ca-cert\") pod \"e2e-inf-a-078357-openshift-default-777d8b777-mjlh2\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:31.714532 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.714508 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9219d2-e2a1-49a3-9068-cc5234ac746c-istio-envoy\") pod \"e2e-inf-a-078357-openshift-default-777d8b777-mjlh2\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:31.714892 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.714844 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9219d2-e2a1-49a3-9068-cc5234ac746c-istio-podinfo\") pod \"e2e-inf-a-078357-openshift-default-777d8b777-mjlh2\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:31.724059 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.723999 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9219d2-e2a1-49a3-9068-cc5234ac746c-istio-token\") pod \"e2e-inf-a-078357-openshift-default-777d8b777-mjlh2\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:31.724648 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.724612 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsqzd\" (UniqueName: \"kubernetes.io/projected/fd9219d2-e2a1-49a3-9068-cc5234ac746c-kube-api-access-nsqzd\") pod \"e2e-inf-a-078357-openshift-default-777d8b777-mjlh2\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:31.840945 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.840839 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:31.993400 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.993369 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2"] Jul 06 21:30:31.995770 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:30:31.995741 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd9219d2_e2a1_49a3_9068_cc5234ac746c.slice/crio-8401a40e49112719944a2855fe305825dc4881c40b4a0fc9177c73961fdaf463 WatchSource:0}: Error finding container 8401a40e49112719944a2855fe305825dc4881c40b4a0fc9177c73961fdaf463: Status 404 returned error can't find the container with id 8401a40e49112719944a2855fe305825dc4881c40b4a0fc9177c73961fdaf463 Jul 06 21:30:31.998112 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.998080 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:30:31.998186 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.998151 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:30:31.998186 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:31.998181 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:30:32.832457 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:32.832410 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" event={"ID":"fd9219d2-e2a1-49a3-9068-cc5234ac746c","Type":"ContainerStarted","Data":"89f9f7b30fed84c0f5d72f6bae420f14cbc184ff4ed3fb4b630445b1f0856776"} Jul 06 21:30:32.832828 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:32.832504 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" event={"ID":"fd9219d2-e2a1-49a3-9068-cc5234ac746c","Type":"ContainerStarted","Data":"8401a40e49112719944a2855fe305825dc4881c40b4a0fc9177c73961fdaf463"} Jul 06 21:30:32.842051 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:32.841975 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:32.856896 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:32.856831 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" podStartSLOduration=1.856810448 podStartE2EDuration="1.856810448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:30:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:30:32.856529937 +0000 UTC m=+1917.174753197" watchObservedRunningTime="2026-07-06 21:30:32.856810448 +0000 UTC m=+1917.175033709" Jul 06 21:30:33.847004 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:33.846971 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:34.844274 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:34.844245 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:34.845163 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:34.845140 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:35.055173 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:35.055131 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-078357-5586f58b9-t8ttg"] Jul 06 21:30:35.059770 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:35.059738 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-078357-5586f58b9-t8ttg" Jul 06 21:30:35.062277 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:35.062250 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-078357-serving-cert\"" Jul 06 21:30:35.078632 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:35.078601 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-078357-5586f58b9-t8ttg"] Jul 06 21:30:35.146443 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:35.146406 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5285b4d2-ca52-440c-80e6-03885e66ee08-openshift-service-ca\") pod \"maas-api-e2e-inf-a-078357-5586f58b9-t8ttg\" (UID: \"5285b4d2-ca52-440c-80e6-03885e66ee08\") " pod="opendatahub/maas-api-e2e-inf-a-078357-5586f58b9-t8ttg" Jul 06 21:30:35.146638 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:35.146542 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24dfv\" (UniqueName: \"kubernetes.io/projected/5285b4d2-ca52-440c-80e6-03885e66ee08-kube-api-access-24dfv\") pod \"maas-api-e2e-inf-a-078357-5586f58b9-t8ttg\" (UID: \"5285b4d2-ca52-440c-80e6-03885e66ee08\") " pod="opendatahub/maas-api-e2e-inf-a-078357-5586f58b9-t8ttg" Jul 06 21:30:35.146638 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:35.146592 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5285b4d2-ca52-440c-80e6-03885e66ee08-maas-api-tls\") pod \"maas-api-e2e-inf-a-078357-5586f58b9-t8ttg\" (UID: \"5285b4d2-ca52-440c-80e6-03885e66ee08\") " pod="opendatahub/maas-api-e2e-inf-a-078357-5586f58b9-t8ttg" Jul 06 21:30:35.248158 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:35.248112 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24dfv\" (UniqueName: \"kubernetes.io/projected/5285b4d2-ca52-440c-80e6-03885e66ee08-kube-api-access-24dfv\") pod \"maas-api-e2e-inf-a-078357-5586f58b9-t8ttg\" (UID: \"5285b4d2-ca52-440c-80e6-03885e66ee08\") " pod="opendatahub/maas-api-e2e-inf-a-078357-5586f58b9-t8ttg" Jul 06 21:30:35.248382 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:35.248174 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5285b4d2-ca52-440c-80e6-03885e66ee08-maas-api-tls\") pod \"maas-api-e2e-inf-a-078357-5586f58b9-t8ttg\" (UID: \"5285b4d2-ca52-440c-80e6-03885e66ee08\") " pod="opendatahub/maas-api-e2e-inf-a-078357-5586f58b9-t8ttg" Jul 06 21:30:35.248382 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:35.248207 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5285b4d2-ca52-440c-80e6-03885e66ee08-openshift-service-ca\") pod \"maas-api-e2e-inf-a-078357-5586f58b9-t8ttg\" (UID: \"5285b4d2-ca52-440c-80e6-03885e66ee08\") " pod="opendatahub/maas-api-e2e-inf-a-078357-5586f58b9-t8ttg" Jul 06 21:30:35.248382 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:30:35.248325 2577 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-a-078357-serving-cert: secret "maas-api-e2e-inf-a-078357-serving-cert" not found Jul 06 21:30:35.248558 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:30:35.248401 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5285b4d2-ca52-440c-80e6-03885e66ee08-maas-api-tls podName:5285b4d2-ca52-440c-80e6-03885e66ee08 nodeName:}" failed. No retries permitted until 2026-07-06 21:30:35.748379717 +0000 UTC m=+1920.066602962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5285b4d2-ca52-440c-80e6-03885e66ee08-maas-api-tls") pod "maas-api-e2e-inf-a-078357-5586f58b9-t8ttg" (UID: "5285b4d2-ca52-440c-80e6-03885e66ee08") : secret "maas-api-e2e-inf-a-078357-serving-cert" not found Jul 06 21:30:35.248827 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:35.248809 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5285b4d2-ca52-440c-80e6-03885e66ee08-openshift-service-ca\") pod \"maas-api-e2e-inf-a-078357-5586f58b9-t8ttg\" (UID: \"5285b4d2-ca52-440c-80e6-03885e66ee08\") " pod="opendatahub/maas-api-e2e-inf-a-078357-5586f58b9-t8ttg" Jul 06 21:30:35.261883 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:35.261849 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24dfv\" (UniqueName: \"kubernetes.io/projected/5285b4d2-ca52-440c-80e6-03885e66ee08-kube-api-access-24dfv\") pod \"maas-api-e2e-inf-a-078357-5586f58b9-t8ttg\" (UID: \"5285b4d2-ca52-440c-80e6-03885e66ee08\") " pod="opendatahub/maas-api-e2e-inf-a-078357-5586f58b9-t8ttg" Jul 06 21:30:35.754080 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:35.754038 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5285b4d2-ca52-440c-80e6-03885e66ee08-maas-api-tls\") pod \"maas-api-e2e-inf-a-078357-5586f58b9-t8ttg\" (UID: \"5285b4d2-ca52-440c-80e6-03885e66ee08\") " pod="opendatahub/maas-api-e2e-inf-a-078357-5586f58b9-t8ttg" Jul 06 21:30:35.754293 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:30:35.754207 2577 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-a-078357-serving-cert: secret "maas-api-e2e-inf-a-078357-serving-cert" not found Jul 06 21:30:35.754351 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:30:35.754299 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5285b4d2-ca52-440c-80e6-03885e66ee08-maas-api-tls podName:5285b4d2-ca52-440c-80e6-03885e66ee08 nodeName:}" failed. No retries permitted until 2026-07-06 21:30:36.754274199 +0000 UTC m=+1921.072497460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5285b4d2-ca52-440c-80e6-03885e66ee08-maas-api-tls") pod "maas-api-e2e-inf-a-078357-5586f58b9-t8ttg" (UID: "5285b4d2-ca52-440c-80e6-03885e66ee08") : secret "maas-api-e2e-inf-a-078357-serving-cert" not found Jul 06 21:30:36.766348 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:36.766232 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5285b4d2-ca52-440c-80e6-03885e66ee08-maas-api-tls\") pod \"maas-api-e2e-inf-a-078357-5586f58b9-t8ttg\" (UID: \"5285b4d2-ca52-440c-80e6-03885e66ee08\") " pod="opendatahub/maas-api-e2e-inf-a-078357-5586f58b9-t8ttg" Jul 06 21:30:36.768797 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:36.768765 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5285b4d2-ca52-440c-80e6-03885e66ee08-maas-api-tls\") pod \"maas-api-e2e-inf-a-078357-5586f58b9-t8ttg\" (UID: \"5285b4d2-ca52-440c-80e6-03885e66ee08\") " pod="opendatahub/maas-api-e2e-inf-a-078357-5586f58b9-t8ttg" Jul 06 21:30:36.873114 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:36.873070 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-078357-5586f58b9-t8ttg" Jul 06 21:30:37.035267 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:37.035228 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-078357-5586f58b9-t8ttg"] Jul 06 21:30:37.036986 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:30:37.036957 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5285b4d2_ca52_440c_80e6_03885e66ee08.slice/crio-c04c103228ecac61a7c895e451f6bc6c114d345dc272021e110d04c9fc295234 WatchSource:0}: Error finding container c04c103228ecac61a7c895e451f6bc6c114d345dc272021e110d04c9fc295234: Status 404 returned error can't find the container with id c04c103228ecac61a7c895e451f6bc6c114d345dc272021e110d04c9fc295234 Jul 06 21:30:37.038622 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:37.038596 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 21:30:37.862803 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:37.862708 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-078357-5586f58b9-t8ttg" event={"ID":"5285b4d2-ca52-440c-80e6-03885e66ee08","Type":"ContainerStarted","Data":"e8977566241825cba050f8053f84eb28e5fdb254332caac1baef3ba587ff8be7"} Jul 06 21:30:37.862803 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:37.862772 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-078357-5586f58b9-t8ttg" event={"ID":"5285b4d2-ca52-440c-80e6-03885e66ee08","Type":"ContainerStarted","Data":"c04c103228ecac61a7c895e451f6bc6c114d345dc272021e110d04c9fc295234"} Jul 06 21:30:37.863244 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:37.862826 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-078357-5586f58b9-t8ttg" Jul 06 21:30:37.885303 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:37.885243 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-078357-5586f58b9-t8ttg" podStartSLOduration=2.31119733 podStartE2EDuration="2.885225864s" totalImagesPullingTime="574.028534ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:30:37.883126032 +0000 UTC m=+1922.201349295" watchObservedRunningTime="2026-07-06 21:30:37.885225864 +0000 UTC m=+1922.203449124" Jul 06 21:30:40.356926 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.356890 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj"] Jul 06 21:30:40.360766 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.360740 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:40.363293 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.363266 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c3d033-openshift-default-dockercfg-vf4bt\"" Jul 06 21:30:40.375969 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.375932 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj"] Jul 06 21:30:40.402217 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.402178 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d4da94-ad69-4461-8572-3f50bf41515b-istiod-ca-cert\") pod \"e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:40.402419 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.402228 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d4da94-ad69-4461-8572-3f50bf41515b-credential-socket\") pod \"e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:40.402419 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.402252 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d4da94-ad69-4461-8572-3f50bf41515b-istio-data\") pod \"e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:40.402419 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.402320 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d4da94-ad69-4461-8572-3f50bf41515b-workload-socket\") pod \"e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:40.402419 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.402349 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhplr\" (UniqueName: \"kubernetes.io/projected/25d4da94-ad69-4461-8572-3f50bf41515b-kube-api-access-mhplr\") pod \"e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:40.402419 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.402383 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d4da94-ad69-4461-8572-3f50bf41515b-workload-certs\") pod \"e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:40.402419 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.402411 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d4da94-ad69-4461-8572-3f50bf41515b-istio-token\") pod \"e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:40.402644 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.402440 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d4da94-ad69-4461-8572-3f50bf41515b-istio-podinfo\") pod \"e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:40.402644 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.402465 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d4da94-ad69-4461-8572-3f50bf41515b-istio-envoy\") pod \"e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:40.503967 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.503910 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d4da94-ad69-4461-8572-3f50bf41515b-istio-podinfo\") pod \"e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:40.503967 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.503971 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d4da94-ad69-4461-8572-3f50bf41515b-istio-envoy\") pod \"e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:40.504276 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.504070 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d4da94-ad69-4461-8572-3f50bf41515b-istiod-ca-cert\") pod \"e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:40.504276 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.504090 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d4da94-ad69-4461-8572-3f50bf41515b-credential-socket\") pod \"e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:40.504276 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.504109 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d4da94-ad69-4461-8572-3f50bf41515b-istio-data\") pod \"e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:40.504276 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.504149 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d4da94-ad69-4461-8572-3f50bf41515b-workload-socket\") pod \"e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:40.504528 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.504466 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhplr\" (UniqueName: \"kubernetes.io/projected/25d4da94-ad69-4461-8572-3f50bf41515b-kube-api-access-mhplr\") pod \"e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:40.504588 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.504560 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d4da94-ad69-4461-8572-3f50bf41515b-workload-certs\") pod \"e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:40.504645 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.504495 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d4da94-ad69-4461-8572-3f50bf41515b-istio-data\") pod \"e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:40.504645 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.504607 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d4da94-ad69-4461-8572-3f50bf41515b-credential-socket\") pod \"e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:40.504645 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.504627 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d4da94-ad69-4461-8572-3f50bf41515b-istio-token\") pod \"e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:40.504877 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.504856 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d4da94-ad69-4461-8572-3f50bf41515b-workload-certs\") pod \"e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:40.504953 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.504681 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d4da94-ad69-4461-8572-3f50bf41515b-workload-socket\") pod \"e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:40.504953 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.504929 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d4da94-ad69-4461-8572-3f50bf41515b-istiod-ca-cert\") pod \"e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:40.506465 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.506443 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d4da94-ad69-4461-8572-3f50bf41515b-istio-envoy\") pod \"e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:40.506779 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.506758 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d4da94-ad69-4461-8572-3f50bf41515b-istio-podinfo\") pod \"e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:40.513270 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.513242 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d4da94-ad69-4461-8572-3f50bf41515b-istio-token\") pod \"e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:40.513402 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.513246 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhplr\" (UniqueName: \"kubernetes.io/projected/25d4da94-ad69-4461-8572-3f50bf41515b-kube-api-access-mhplr\") pod \"e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:40.672949 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.672908 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:40.843440 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.843411 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj"] Jul 06 21:30:40.846124 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:30:40.846086 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d4da94_ad69_4461_8572_3f50bf41515b.slice/crio-199d77e65764b05c6ab194540a502745e33c30b83718d2ae19cf752ffbcd4a41 WatchSource:0}: Error finding container 199d77e65764b05c6ab194540a502745e33c30b83718d2ae19cf752ffbcd4a41: Status 404 returned error can't find the container with id 199d77e65764b05c6ab194540a502745e33c30b83718d2ae19cf752ffbcd4a41 Jul 06 21:30:40.848386 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.848352 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:30:40.848467 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.848420 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:30:40.848467 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.848451 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:30:40.878077 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:40.878045 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" event={"ID":"25d4da94-ad69-4461-8572-3f50bf41515b","Type":"ContainerStarted","Data":"199d77e65764b05c6ab194540a502745e33c30b83718d2ae19cf752ffbcd4a41"} Jul 06 21:30:41.884958 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:41.884919 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" event={"ID":"25d4da94-ad69-4461-8572-3f50bf41515b","Type":"ContainerStarted","Data":"13eb914e9335308184ae5121fbe4aa8b3f501c32991ca807552dc1c94e7baa55"} Jul 06 21:30:41.912244 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:41.912189 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" podStartSLOduration=1.912170219 podStartE2EDuration="1.912170219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:30:41.908396667 +0000 UTC m=+1926.226619927" watchObservedRunningTime="2026-07-06 21:30:41.912170219 +0000 UTC m=+1926.230393478" Jul 06 21:30:42.673400 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:42.673355 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:42.678507 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:42.678478 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:42.890845 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:42.890809 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:42.891771 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:42.891749 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:30:43.873485 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:43.873453 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-078357-5586f58b9-t8ttg" Jul 06 21:30:44.152411 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:44.152315 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc"] Jul 06 21:30:44.157088 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:44.157049 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc" Jul 06 21:30:44.160086 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:44.160059 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-c3d033-serving-cert\"" Jul 06 21:30:44.179459 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:44.179370 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc"] Jul 06 21:30:44.257831 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:44.257787 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e07384be-63e9-49c5-9fad-626e0ad01c71-maas-api-tls\") pod \"maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc\" (UID: \"e07384be-63e9-49c5-9fad-626e0ad01c71\") " pod="opendatahub/maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc" Jul 06 21:30:44.258038 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:44.257931 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e07384be-63e9-49c5-9fad-626e0ad01c71-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc\" (UID: \"e07384be-63e9-49c5-9fad-626e0ad01c71\") " pod="opendatahub/maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc" Jul 06 21:30:44.258111 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:44.258029 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6gj4\" (UniqueName: \"kubernetes.io/projected/e07384be-63e9-49c5-9fad-626e0ad01c71-kube-api-access-t6gj4\") pod \"maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc\" (UID: \"e07384be-63e9-49c5-9fad-626e0ad01c71\") " pod="opendatahub/maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc" Jul 06 21:30:44.358993 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:44.358949 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6gj4\" (UniqueName: \"kubernetes.io/projected/e07384be-63e9-49c5-9fad-626e0ad01c71-kube-api-access-t6gj4\") pod \"maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc\" (UID: \"e07384be-63e9-49c5-9fad-626e0ad01c71\") " pod="opendatahub/maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc" Jul 06 21:30:44.359233 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:44.359086 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e07384be-63e9-49c5-9fad-626e0ad01c71-maas-api-tls\") pod \"maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc\" (UID: \"e07384be-63e9-49c5-9fad-626e0ad01c71\") " pod="opendatahub/maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc" Jul 06 21:30:44.359233 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:44.359182 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e07384be-63e9-49c5-9fad-626e0ad01c71-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc\" (UID: \"e07384be-63e9-49c5-9fad-626e0ad01c71\") " pod="opendatahub/maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc" Jul 06 21:30:44.359233 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:30:44.359220 2577 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-b-c3d033-serving-cert: secret "maas-api-e2e-inf-b-c3d033-serving-cert" not found Jul 06 21:30:44.359431 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:30:44.359285 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e07384be-63e9-49c5-9fad-626e0ad01c71-maas-api-tls podName:e07384be-63e9-49c5-9fad-626e0ad01c71 nodeName:}" failed. No retries permitted until 2026-07-06 21:30:44.859265824 +0000 UTC m=+1929.177489074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e07384be-63e9-49c5-9fad-626e0ad01c71-maas-api-tls") pod "maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc" (UID: "e07384be-63e9-49c5-9fad-626e0ad01c71") : secret "maas-api-e2e-inf-b-c3d033-serving-cert" not found Jul 06 21:30:44.359902 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:44.359877 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e07384be-63e9-49c5-9fad-626e0ad01c71-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc\" (UID: \"e07384be-63e9-49c5-9fad-626e0ad01c71\") " pod="opendatahub/maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc" Jul 06 21:30:44.369202 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:44.369167 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6gj4\" (UniqueName: \"kubernetes.io/projected/e07384be-63e9-49c5-9fad-626e0ad01c71-kube-api-access-t6gj4\") pod \"maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc\" (UID: \"e07384be-63e9-49c5-9fad-626e0ad01c71\") " pod="opendatahub/maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc" Jul 06 21:30:44.864566 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:44.864521 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e07384be-63e9-49c5-9fad-626e0ad01c71-maas-api-tls\") pod \"maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc\" (UID: \"e07384be-63e9-49c5-9fad-626e0ad01c71\") " pod="opendatahub/maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc" Jul 06 21:30:44.867177 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:44.867154 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e07384be-63e9-49c5-9fad-626e0ad01c71-maas-api-tls\") pod \"maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc\" (UID: \"e07384be-63e9-49c5-9fad-626e0ad01c71\") " pod="opendatahub/maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc" Jul 06 21:30:45.074854 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:45.074808 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc" Jul 06 21:30:45.277046 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:45.276995 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc"] Jul 06 21:30:45.278970 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:30:45.278918 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode07384be_63e9_49c5_9fad_626e0ad01c71.slice/crio-bba26578e8af4190fe2b7ae875c59c75db9a403d17208a3e545ac422fe999bee WatchSource:0}: Error finding container bba26578e8af4190fe2b7ae875c59c75db9a403d17208a3e545ac422fe999bee: Status 404 returned error can't find the container with id bba26578e8af4190fe2b7ae875c59c75db9a403d17208a3e545ac422fe999bee Jul 06 21:30:45.909796 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:45.909762 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc" event={"ID":"e07384be-63e9-49c5-9fad-626e0ad01c71","Type":"ContainerStarted","Data":"761c6f9c7156dcf1925fdcfa1822a3ce952124e1ef44bb5255409c6170981151"} Jul 06 21:30:45.909977 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:45.909817 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc" event={"ID":"e07384be-63e9-49c5-9fad-626e0ad01c71","Type":"ContainerStarted","Data":"bba26578e8af4190fe2b7ae875c59c75db9a403d17208a3e545ac422fe999bee"} Jul 06 21:30:45.909977 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:45.909908 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc" Jul 06 21:30:45.939390 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:45.939251 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc" podStartSLOduration=1.538939433 podStartE2EDuration="1.939221984s" totalImagesPullingTime="400.282551ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:30:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:30:45.928105305 +0000 UTC m=+1930.246328567" watchObservedRunningTime="2026-07-06 21:30:45.939221984 +0000 UTC m=+1930.257445245" Jul 06 21:30:49.846335 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:49.846295 2577 patch_prober.go:28] interesting pod/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 06 21:30:49.846785 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:49.846354 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" podUID="fd9219d2-e2a1-49a3-9068-cc5234ac746c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 06 21:30:49.934675 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:49.934637 2577 generic.go:356] "Generic (PLEG): container finished" podID="fd9219d2-e2a1-49a3-9068-cc5234ac746c" containerID="89f9f7b30fed84c0f5d72f6bae420f14cbc184ff4ed3fb4b630445b1f0856776" exitCode=0 Jul 06 21:30:49.934956 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:49.934713 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" event={"ID":"fd9219d2-e2a1-49a3-9068-cc5234ac746c","Type":"ContainerDied","Data":"89f9f7b30fed84c0f5d72f6bae420f14cbc184ff4ed3fb4b630445b1f0856776"} Jul 06 21:30:49.935258 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:49.935240 2577 scope.go:117] "RemoveContainer" containerID="89f9f7b30fed84c0f5d72f6bae420f14cbc184ff4ed3fb4b630445b1f0856776" Jul 06 21:30:49.936348 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:49.936324 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:30:49.936407 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:49.936391 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:30:49.936443 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:49.936430 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:30:50.946455 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:50.946407 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" event={"ID":"fd9219d2-e2a1-49a3-9068-cc5234ac746c","Type":"ContainerStarted","Data":"9999d085e23add44b55ce449119e28740003b8a681c6c1c620fad3f55c548871"} Jul 06 21:30:51.842195 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:51.842144 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:51.842566 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:51.842308 2577 patch_prober.go:28] interesting pod/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 06 21:30:51.842810 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:51.842776 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" podUID="fd9219d2-e2a1-49a3-9068-cc5234ac746c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 06 21:30:51.921590 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:51.921562 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc" Jul 06 21:30:52.842454 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:52.842405 2577 patch_prober.go:28] interesting pod/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 06 21:30:52.842975 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:52.842494 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" podUID="fd9219d2-e2a1-49a3-9068-cc5234ac746c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 06 21:30:53.842322 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:53.842269 2577 patch_prober.go:28] interesting pod/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 06 21:30:53.842562 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:53.842387 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" podUID="fd9219d2-e2a1-49a3-9068-cc5234ac746c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 06 21:30:54.841740 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:54.841699 2577 patch_prober.go:28] interesting pod/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 06 21:30:54.842114 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:54.842081 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" podUID="fd9219d2-e2a1-49a3-9068-cc5234ac746c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 06 21:30:55.846181 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:55.846149 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:55.846658 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:55.846430 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:55.847313 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:55.847296 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:30:58.016318 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:58.016181 2577 patch_prober.go:28] interesting pod/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": read tcp 10.133.0.2:38258->10.133.0.58:15021: read: connection reset by peer" start-of-body= Jul 06 21:30:58.016318 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:58.016248 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" podUID="25d4da94-ad69-4461-8572-3f50bf41515b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": read tcp 10.133.0.2:38258->10.133.0.58:15021: read: connection reset by peer" Jul 06 21:30:59.001093 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:59.001054 2577 generic.go:356] "Generic (PLEG): container finished" podID="25d4da94-ad69-4461-8572-3f50bf41515b" containerID="13eb914e9335308184ae5121fbe4aa8b3f501c32991ca807552dc1c94e7baa55" exitCode=0 Jul 06 21:30:59.001286 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:59.001126 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" event={"ID":"25d4da94-ad69-4461-8572-3f50bf41515b","Type":"ContainerDied","Data":"13eb914e9335308184ae5121fbe4aa8b3f501c32991ca807552dc1c94e7baa55"} Jul 06 21:30:59.001569 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:59.001554 2577 scope.go:117] "RemoveContainer" containerID="13eb914e9335308184ae5121fbe4aa8b3f501c32991ca807552dc1c94e7baa55" Jul 06 21:30:59.002642 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:59.002614 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:30:59.002706 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:59.002685 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:30:59.002754 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:30:59.002717 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:31:00.008727 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:31:00.008686 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" event={"ID":"25d4da94-ad69-4461-8572-3f50bf41515b","Type":"ContainerStarted","Data":"e9143525f043e1ce57274e48cc3c4fc85473e3643a0c51141e8af73d3433ef8b"} Jul 06 21:31:00.673914 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:31:00.673869 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:31:00.674173 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:31:00.674148 2577 patch_prober.go:28] interesting pod/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 06 21:31:00.674226 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:31:00.674199 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" podUID="25d4da94-ad69-4461-8572-3f50bf41515b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 06 21:31:01.673878 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:31:01.673839 2577 patch_prober.go:28] interesting pod/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 06 21:31:01.674305 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:31:01.673897 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" podUID="25d4da94-ad69-4461-8572-3f50bf41515b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 06 21:31:02.673340 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:31:02.673302 2577 patch_prober.go:28] interesting pod/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 06 21:31:02.673537 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:31:02.673358 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" podUID="25d4da94-ad69-4461-8572-3f50bf41515b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 06 21:31:03.674119 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:31:03.674079 2577 patch_prober.go:28] interesting pod/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 06 21:31:03.674542 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:31:03.674134 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" podUID="25d4da94-ad69-4461-8572-3f50bf41515b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 06 21:31:04.676540 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:31:04.676507 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:31:04.677058 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:31:04.677035 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:31:04.677929 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:31:04.677905 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:31:59.855884 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:31:59.855780 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-078357-5586f58b9-t8ttg"] Jul 06 21:31:59.856543 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:31:59.856032 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-078357-5586f58b9-t8ttg" podUID="5285b4d2-ca52-440c-80e6-03885e66ee08" containerName="maas-api" containerID="cri-o://e8977566241825cba050f8053f84eb28e5fdb254332caac1baef3ba587ff8be7" gracePeriod=30 Jul 06 21:32:00.109765 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:00.109692 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-078357-5586f58b9-t8ttg" Jul 06 21:32:00.235239 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:00.235197 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24dfv\" (UniqueName: \"kubernetes.io/projected/5285b4d2-ca52-440c-80e6-03885e66ee08-kube-api-access-24dfv\") pod \"5285b4d2-ca52-440c-80e6-03885e66ee08\" (UID: \"5285b4d2-ca52-440c-80e6-03885e66ee08\") " Jul 06 21:32:00.235429 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:00.235346 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5285b4d2-ca52-440c-80e6-03885e66ee08-openshift-service-ca\") pod \"5285b4d2-ca52-440c-80e6-03885e66ee08\" (UID: \"5285b4d2-ca52-440c-80e6-03885e66ee08\") " Jul 06 21:32:00.235429 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:00.235395 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5285b4d2-ca52-440c-80e6-03885e66ee08-maas-api-tls\") pod \"5285b4d2-ca52-440c-80e6-03885e66ee08\" (UID: \"5285b4d2-ca52-440c-80e6-03885e66ee08\") " Jul 06 21:32:00.235706 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:00.235677 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5285b4d2-ca52-440c-80e6-03885e66ee08-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5285b4d2-ca52-440c-80e6-03885e66ee08" (UID: "5285b4d2-ca52-440c-80e6-03885e66ee08"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:32:00.237540 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:00.237509 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5285b4d2-ca52-440c-80e6-03885e66ee08-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5285b4d2-ca52-440c-80e6-03885e66ee08" (UID: "5285b4d2-ca52-440c-80e6-03885e66ee08"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:32:00.237602 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:00.237581 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5285b4d2-ca52-440c-80e6-03885e66ee08-kube-api-access-24dfv" (OuterVolumeSpecName: "kube-api-access-24dfv") pod "5285b4d2-ca52-440c-80e6-03885e66ee08" (UID: "5285b4d2-ca52-440c-80e6-03885e66ee08"). InnerVolumeSpecName "kube-api-access-24dfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:32:00.336667 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:00.336627 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24dfv\" (UniqueName: \"kubernetes.io/projected/5285b4d2-ca52-440c-80e6-03885e66ee08-kube-api-access-24dfv\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:32:00.336667 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:00.336663 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5285b4d2-ca52-440c-80e6-03885e66ee08-openshift-service-ca\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:32:00.336890 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:00.336677 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5285b4d2-ca52-440c-80e6-03885e66ee08-maas-api-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:32:00.346545 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:00.346509 2577 generic.go:356] "Generic (PLEG): container finished" podID="5285b4d2-ca52-440c-80e6-03885e66ee08" containerID="e8977566241825cba050f8053f84eb28e5fdb254332caac1baef3ba587ff8be7" exitCode=0 Jul 06 21:32:00.346713 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:00.346566 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-078357-5586f58b9-t8ttg" Jul 06 21:32:00.346713 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:00.346580 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-078357-5586f58b9-t8ttg" event={"ID":"5285b4d2-ca52-440c-80e6-03885e66ee08","Type":"ContainerDied","Data":"e8977566241825cba050f8053f84eb28e5fdb254332caac1baef3ba587ff8be7"} Jul 06 21:32:00.346713 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:00.346630 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-078357-5586f58b9-t8ttg" event={"ID":"5285b4d2-ca52-440c-80e6-03885e66ee08","Type":"ContainerDied","Data":"c04c103228ecac61a7c895e451f6bc6c114d345dc272021e110d04c9fc295234"} Jul 06 21:32:00.346713 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:00.346654 2577 scope.go:117] "RemoveContainer" containerID="e8977566241825cba050f8053f84eb28e5fdb254332caac1baef3ba587ff8be7" Jul 06 21:32:00.356580 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:00.356559 2577 scope.go:117] "RemoveContainer" containerID="e8977566241825cba050f8053f84eb28e5fdb254332caac1baef3ba587ff8be7" Jul 06 21:32:00.356924 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:32:00.356905 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8977566241825cba050f8053f84eb28e5fdb254332caac1baef3ba587ff8be7\": container with ID starting with e8977566241825cba050f8053f84eb28e5fdb254332caac1baef3ba587ff8be7 not found: ID does not exist" containerID="e8977566241825cba050f8053f84eb28e5fdb254332caac1baef3ba587ff8be7" Jul 06 21:32:00.356969 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:00.356943 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8977566241825cba050f8053f84eb28e5fdb254332caac1baef3ba587ff8be7"} err="failed to get container status \"e8977566241825cba050f8053f84eb28e5fdb254332caac1baef3ba587ff8be7\": rpc error: code = NotFound desc = could not find container \"e8977566241825cba050f8053f84eb28e5fdb254332caac1baef3ba587ff8be7\": container with ID starting with e8977566241825cba050f8053f84eb28e5fdb254332caac1baef3ba587ff8be7 not found: ID does not exist" Jul 06 21:32:00.370291 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:00.370197 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-078357-5586f58b9-t8ttg"] Jul 06 21:32:00.375305 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:00.375271 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-078357-5586f58b9-t8ttg"] Jul 06 21:32:02.169597 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:02.169561 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5285b4d2-ca52-440c-80e6-03885e66ee08" path="/var/lib/kubelet/pods/5285b4d2-ca52-440c-80e6-03885e66ee08/volumes" Jul 06 21:32:42.328865 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:42.328824 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2"] Jul 06 21:32:42.329732 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:42.329693 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" podUID="fd9219d2-e2a1-49a3-9068-cc5234ac746c" containerName="istio-proxy" containerID="cri-o://9999d085e23add44b55ce449119e28740003b8a681c6c1c620fad3f55c548871" gracePeriod=30 Jul 06 21:32:43.552793 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:43.552748 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc"] Jul 06 21:32:43.553310 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:43.553037 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc" podUID="e07384be-63e9-49c5-9fad-626e0ad01c71" containerName="maas-api" containerID="cri-o://761c6f9c7156dcf1925fdcfa1822a3ce952124e1ef44bb5255409c6170981151" gracePeriod=30 Jul 06 21:32:43.809386 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:43.809313 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc" Jul 06 21:32:43.967321 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:43.967274 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e07384be-63e9-49c5-9fad-626e0ad01c71-openshift-service-ca\") pod \"e07384be-63e9-49c5-9fad-626e0ad01c71\" (UID: \"e07384be-63e9-49c5-9fad-626e0ad01c71\") " Jul 06 21:32:43.967530 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:43.967338 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e07384be-63e9-49c5-9fad-626e0ad01c71-maas-api-tls\") pod \"e07384be-63e9-49c5-9fad-626e0ad01c71\" (UID: \"e07384be-63e9-49c5-9fad-626e0ad01c71\") " Jul 06 21:32:43.967530 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:43.967358 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6gj4\" (UniqueName: \"kubernetes.io/projected/e07384be-63e9-49c5-9fad-626e0ad01c71-kube-api-access-t6gj4\") pod \"e07384be-63e9-49c5-9fad-626e0ad01c71\" (UID: \"e07384be-63e9-49c5-9fad-626e0ad01c71\") " Jul 06 21:32:43.967740 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:43.967716 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e07384be-63e9-49c5-9fad-626e0ad01c71-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e07384be-63e9-49c5-9fad-626e0ad01c71" (UID: "e07384be-63e9-49c5-9fad-626e0ad01c71"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:32:43.969540 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:43.969506 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e07384be-63e9-49c5-9fad-626e0ad01c71-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e07384be-63e9-49c5-9fad-626e0ad01c71" (UID: "e07384be-63e9-49c5-9fad-626e0ad01c71"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:32:43.969652 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:43.969536 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e07384be-63e9-49c5-9fad-626e0ad01c71-kube-api-access-t6gj4" (OuterVolumeSpecName: "kube-api-access-t6gj4") pod "e07384be-63e9-49c5-9fad-626e0ad01c71" (UID: "e07384be-63e9-49c5-9fad-626e0ad01c71"). InnerVolumeSpecName "kube-api-access-t6gj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:32:44.068616 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:44.068518 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e07384be-63e9-49c5-9fad-626e0ad01c71-openshift-service-ca\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:32:44.068616 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:44.068551 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e07384be-63e9-49c5-9fad-626e0ad01c71-maas-api-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:32:44.068616 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:44.068561 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6gj4\" (UniqueName: \"kubernetes.io/projected/e07384be-63e9-49c5-9fad-626e0ad01c71-kube-api-access-t6gj4\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:32:44.599160 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:44.599122 2577 generic.go:356] "Generic (PLEG): container finished" podID="e07384be-63e9-49c5-9fad-626e0ad01c71" containerID="761c6f9c7156dcf1925fdcfa1822a3ce952124e1ef44bb5255409c6170981151" exitCode=0 Jul 06 21:32:44.599575 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:44.599178 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc" Jul 06 21:32:44.599575 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:44.599201 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc" event={"ID":"e07384be-63e9-49c5-9fad-626e0ad01c71","Type":"ContainerDied","Data":"761c6f9c7156dcf1925fdcfa1822a3ce952124e1ef44bb5255409c6170981151"} Jul 06 21:32:44.599575 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:44.599277 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc" event={"ID":"e07384be-63e9-49c5-9fad-626e0ad01c71","Type":"ContainerDied","Data":"bba26578e8af4190fe2b7ae875c59c75db9a403d17208a3e545ac422fe999bee"} Jul 06 21:32:44.599575 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:44.599299 2577 scope.go:117] "RemoveContainer" containerID="761c6f9c7156dcf1925fdcfa1822a3ce952124e1ef44bb5255409c6170981151" Jul 06 21:32:44.608886 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:44.608866 2577 scope.go:117] "RemoveContainer" containerID="761c6f9c7156dcf1925fdcfa1822a3ce952124e1ef44bb5255409c6170981151" Jul 06 21:32:44.609229 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:32:44.609204 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"761c6f9c7156dcf1925fdcfa1822a3ce952124e1ef44bb5255409c6170981151\": container with ID starting with 761c6f9c7156dcf1925fdcfa1822a3ce952124e1ef44bb5255409c6170981151 not found: ID does not exist" containerID="761c6f9c7156dcf1925fdcfa1822a3ce952124e1ef44bb5255409c6170981151" Jul 06 21:32:44.609278 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:44.609240 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"761c6f9c7156dcf1925fdcfa1822a3ce952124e1ef44bb5255409c6170981151"} err="failed to get container status \"761c6f9c7156dcf1925fdcfa1822a3ce952124e1ef44bb5255409c6170981151\": rpc error: code = NotFound desc = could not find container \"761c6f9c7156dcf1925fdcfa1822a3ce952124e1ef44bb5255409c6170981151\": container with ID starting with 761c6f9c7156dcf1925fdcfa1822a3ce952124e1ef44bb5255409c6170981151 not found: ID does not exist" Jul 06 21:32:44.618460 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:44.618427 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc"] Jul 06 21:32:44.624616 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:44.624580 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-c3d033-5fdd88c656-wfbrc"] Jul 06 21:32:46.169356 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:46.169320 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e07384be-63e9-49c5-9fad-626e0ad01c71" path="/var/lib/kubelet/pods/e07384be-63e9-49c5-9fad-626e0ad01c71/volumes" Jul 06 21:32:47.606815 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.606731 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:32:47.618682 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.618645 2577 generic.go:356] "Generic (PLEG): container finished" podID="fd9219d2-e2a1-49a3-9068-cc5234ac746c" containerID="9999d085e23add44b55ce449119e28740003b8a681c6c1c620fad3f55c548871" exitCode=0 Jul 06 21:32:47.618682 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.618691 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" event={"ID":"fd9219d2-e2a1-49a3-9068-cc5234ac746c","Type":"ContainerDied","Data":"9999d085e23add44b55ce449119e28740003b8a681c6c1c620fad3f55c548871"} Jul 06 21:32:47.618959 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.618706 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" Jul 06 21:32:47.618959 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.618750 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2" event={"ID":"fd9219d2-e2a1-49a3-9068-cc5234ac746c","Type":"ContainerDied","Data":"8401a40e49112719944a2855fe305825dc4881c40b4a0fc9177c73961fdaf463"} Jul 06 21:32:47.618959 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.618782 2577 scope.go:117] "RemoveContainer" containerID="9999d085e23add44b55ce449119e28740003b8a681c6c1c620fad3f55c548871" Jul 06 21:32:47.628498 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.628359 2577 scope.go:117] "RemoveContainer" containerID="89f9f7b30fed84c0f5d72f6bae420f14cbc184ff4ed3fb4b630445b1f0856776" Jul 06 21:32:47.641232 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.641193 2577 scope.go:117] "RemoveContainer" containerID="9999d085e23add44b55ce449119e28740003b8a681c6c1c620fad3f55c548871" Jul 06 21:32:47.641602 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:32:47.641582 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9999d085e23add44b55ce449119e28740003b8a681c6c1c620fad3f55c548871\": container with ID starting with 9999d085e23add44b55ce449119e28740003b8a681c6c1c620fad3f55c548871 not found: ID does not exist" containerID="9999d085e23add44b55ce449119e28740003b8a681c6c1c620fad3f55c548871" Jul 06 21:32:47.641663 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.641615 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9999d085e23add44b55ce449119e28740003b8a681c6c1c620fad3f55c548871"} err="failed to get container status \"9999d085e23add44b55ce449119e28740003b8a681c6c1c620fad3f55c548871\": rpc error: code = NotFound desc = could not find container \"9999d085e23add44b55ce449119e28740003b8a681c6c1c620fad3f55c548871\": container with ID starting with 9999d085e23add44b55ce449119e28740003b8a681c6c1c620fad3f55c548871 not found: ID does not exist" Jul 06 21:32:47.641663 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.641633 2577 scope.go:117] "RemoveContainer" containerID="89f9f7b30fed84c0f5d72f6bae420f14cbc184ff4ed3fb4b630445b1f0856776" Jul 06 21:32:47.641879 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:32:47.641860 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89f9f7b30fed84c0f5d72f6bae420f14cbc184ff4ed3fb4b630445b1f0856776\": container with ID starting with 89f9f7b30fed84c0f5d72f6bae420f14cbc184ff4ed3fb4b630445b1f0856776 not found: ID does not exist" containerID="89f9f7b30fed84c0f5d72f6bae420f14cbc184ff4ed3fb4b630445b1f0856776" Jul 06 21:32:47.641957 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.641890 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89f9f7b30fed84c0f5d72f6bae420f14cbc184ff4ed3fb4b630445b1f0856776"} err="failed to get container status \"89f9f7b30fed84c0f5d72f6bae420f14cbc184ff4ed3fb4b630445b1f0856776\": rpc error: code = NotFound desc = could not find container \"89f9f7b30fed84c0f5d72f6bae420f14cbc184ff4ed3fb4b630445b1f0856776\": container with ID starting with 89f9f7b30fed84c0f5d72f6bae420f14cbc184ff4ed3fb4b630445b1f0856776 not found: ID does not exist" Jul 06 21:32:47.704594 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.704551 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9219d2-e2a1-49a3-9068-cc5234ac746c-istio-envoy\") pod \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " Jul 06 21:32:47.704813 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.704660 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9219d2-e2a1-49a3-9068-cc5234ac746c-istio-podinfo\") pod \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " Jul 06 21:32:47.704813 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.704710 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9219d2-e2a1-49a3-9068-cc5234ac746c-istiod-ca-cert\") pod \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " Jul 06 21:32:47.704813 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.704744 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9219d2-e2a1-49a3-9068-cc5234ac746c-workload-socket\") pod \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " Jul 06 21:32:47.704813 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.704770 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9219d2-e2a1-49a3-9068-cc5234ac746c-istio-token\") pod \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " Jul 06 21:32:47.704813 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.704793 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsqzd\" (UniqueName: \"kubernetes.io/projected/fd9219d2-e2a1-49a3-9068-cc5234ac746c-kube-api-access-nsqzd\") pod \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " Jul 06 21:32:47.705138 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.704821 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9219d2-e2a1-49a3-9068-cc5234ac746c-workload-certs\") pod \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " Jul 06 21:32:47.705138 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.704863 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9219d2-e2a1-49a3-9068-cc5234ac746c-istio-data\") pod \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " Jul 06 21:32:47.705138 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.704886 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9219d2-e2a1-49a3-9068-cc5234ac746c-credential-socket\") pod \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\" (UID: \"fd9219d2-e2a1-49a3-9068-cc5234ac746c\") " Jul 06 21:32:47.705138 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.705112 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9219d2-e2a1-49a3-9068-cc5234ac746c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fd9219d2-e2a1-49a3-9068-cc5234ac746c" (UID: "fd9219d2-e2a1-49a3-9068-cc5234ac746c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:32:47.705338 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.705192 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd9219d2-e2a1-49a3-9068-cc5234ac746c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fd9219d2-e2a1-49a3-9068-cc5234ac746c" (UID: "fd9219d2-e2a1-49a3-9068-cc5234ac746c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:32:47.705338 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.705255 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9219d2-e2a1-49a3-9068-cc5234ac746c-workload-socket\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:32:47.705450 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.705334 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9219d2-e2a1-49a3-9068-cc5234ac746c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fd9219d2-e2a1-49a3-9068-cc5234ac746c" (UID: "fd9219d2-e2a1-49a3-9068-cc5234ac746c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:32:47.705545 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.705518 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9219d2-e2a1-49a3-9068-cc5234ac746c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fd9219d2-e2a1-49a3-9068-cc5234ac746c" (UID: "fd9219d2-e2a1-49a3-9068-cc5234ac746c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:32:47.706124 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.706099 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9219d2-e2a1-49a3-9068-cc5234ac746c-istio-data" (OuterVolumeSpecName: "istio-data") pod "fd9219d2-e2a1-49a3-9068-cc5234ac746c" (UID: "fd9219d2-e2a1-49a3-9068-cc5234ac746c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:32:47.707176 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.707150 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9219d2-e2a1-49a3-9068-cc5234ac746c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fd9219d2-e2a1-49a3-9068-cc5234ac746c" (UID: "fd9219d2-e2a1-49a3-9068-cc5234ac746c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:32:47.707446 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.707423 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd9219d2-e2a1-49a3-9068-cc5234ac746c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fd9219d2-e2a1-49a3-9068-cc5234ac746c" (UID: "fd9219d2-e2a1-49a3-9068-cc5234ac746c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 21:32:47.707531 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.707430 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd9219d2-e2a1-49a3-9068-cc5234ac746c-istio-token" (OuterVolumeSpecName: "istio-token") pod "fd9219d2-e2a1-49a3-9068-cc5234ac746c" (UID: "fd9219d2-e2a1-49a3-9068-cc5234ac746c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:32:47.707602 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.707581 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd9219d2-e2a1-49a3-9068-cc5234ac746c-kube-api-access-nsqzd" (OuterVolumeSpecName: "kube-api-access-nsqzd") pod "fd9219d2-e2a1-49a3-9068-cc5234ac746c" (UID: "fd9219d2-e2a1-49a3-9068-cc5234ac746c"). InnerVolumeSpecName "kube-api-access-nsqzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:32:47.806297 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.806259 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9219d2-e2a1-49a3-9068-cc5234ac746c-istio-envoy\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:32:47.806297 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.806290 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9219d2-e2a1-49a3-9068-cc5234ac746c-istio-podinfo\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:32:47.806297 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.806301 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9219d2-e2a1-49a3-9068-cc5234ac746c-istiod-ca-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:32:47.806558 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.806311 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9219d2-e2a1-49a3-9068-cc5234ac746c-istio-token\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:32:47.806558 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.806322 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsqzd\" (UniqueName: \"kubernetes.io/projected/fd9219d2-e2a1-49a3-9068-cc5234ac746c-kube-api-access-nsqzd\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:32:47.806558 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.806330 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9219d2-e2a1-49a3-9068-cc5234ac746c-workload-certs\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:32:47.806558 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.806337 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9219d2-e2a1-49a3-9068-cc5234ac746c-istio-data\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:32:47.806558 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.806346 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9219d2-e2a1-49a3-9068-cc5234ac746c-credential-socket\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:32:47.956342 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.956301 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2"] Jul 06 21:32:47.965191 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:47.965160 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-078357-openshift-default-777d8b777-mjlh2"] Jul 06 21:32:48.169159 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:48.169125 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd9219d2-e2a1-49a3-9068-cc5234ac746c" path="/var/lib/kubelet/pods/fd9219d2-e2a1-49a3-9068-cc5234ac746c/volumes" Jul 06 21:32:49.512839 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:49.512799 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj"] Jul 06 21:32:49.513311 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:49.513088 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" podUID="25d4da94-ad69-4461-8572-3f50bf41515b" containerName="istio-proxy" containerID="cri-o://e9143525f043e1ce57274e48cc3c4fc85473e3643a0c51141e8af73d3433ef8b" gracePeriod=30 Jul 06 21:32:49.679404 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:49.679372 2577 patch_prober.go:28] interesting pod/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 21:32:49.679573 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:49.679429 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" podUID="25d4da94-ad69-4461-8572-3f50bf41515b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 21:32:50.489611 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.489572 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn"] Jul 06 21:32:50.490211 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.490195 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5285b4d2-ca52-440c-80e6-03885e66ee08" containerName="maas-api" Jul 06 21:32:50.490276 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.490214 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5285b4d2-ca52-440c-80e6-03885e66ee08" containerName="maas-api" Jul 06 21:32:50.490276 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.490231 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e07384be-63e9-49c5-9fad-626e0ad01c71" containerName="maas-api" Jul 06 21:32:50.490276 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.490240 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e07384be-63e9-49c5-9fad-626e0ad01c71" containerName="maas-api" Jul 06 21:32:50.490276 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.490262 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd9219d2-e2a1-49a3-9068-cc5234ac746c" containerName="istio-proxy" Jul 06 21:32:50.490276 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.490271 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd9219d2-e2a1-49a3-9068-cc5234ac746c" containerName="istio-proxy" Jul 06 21:32:50.490446 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.490293 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd9219d2-e2a1-49a3-9068-cc5234ac746c" containerName="istio-proxy" Jul 06 21:32:50.490446 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.490301 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd9219d2-e2a1-49a3-9068-cc5234ac746c" containerName="istio-proxy" Jul 06 21:32:50.490446 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.490403 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e07384be-63e9-49c5-9fad-626e0ad01c71" containerName="maas-api" Jul 06 21:32:50.490446 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.490419 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5285b4d2-ca52-440c-80e6-03885e66ee08" containerName="maas-api" Jul 06 21:32:50.490446 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.490429 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd9219d2-e2a1-49a3-9068-cc5234ac746c" containerName="istio-proxy" Jul 06 21:32:50.490446 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.490440 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd9219d2-e2a1-49a3-9068-cc5234ac746c" containerName="istio-proxy" Jul 06 21:32:50.495747 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.495720 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:50.498190 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.498167 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-778c48-openshift-default-dockercfg-h5w8w\"" Jul 06 21:32:50.504837 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.504809 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn"] Jul 06 21:32:50.531300 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.531256 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-istio-envoy\") pod \"e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:50.531300 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.531305 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-workload-socket\") pod \"e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:50.531820 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.531449 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-istio-data\") pod \"e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:50.531820 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.531494 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-istio-token\") pod \"e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:50.531820 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.531516 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-istiod-ca-cert\") pod \"e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:50.531820 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.531546 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-istio-podinfo\") pod \"e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:50.531820 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.531619 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm4nt\" (UniqueName: \"kubernetes.io/projected/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-kube-api-access-wm4nt\") pod \"e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:50.531820 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.531662 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-credential-socket\") pod \"e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:50.531820 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.531677 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-workload-certs\") pod \"e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:50.632616 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.632584 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-istio-data\") pod \"e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:50.632616 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.632620 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-istio-token\") pod \"e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:50.632898 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.632636 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-istiod-ca-cert\") pod \"e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:50.632898 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.632659 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-istio-podinfo\") pod \"e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:50.632898 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.632711 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm4nt\" (UniqueName: \"kubernetes.io/projected/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-kube-api-access-wm4nt\") pod \"e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:50.632898 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.632769 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-credential-socket\") pod \"e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:50.632898 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.632791 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-workload-certs\") pod \"e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:50.632898 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.632822 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-istio-envoy\") pod \"e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:50.632898 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.632854 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-workload-socket\") pod \"e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:50.633373 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.633345 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-workload-certs\") pod \"e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:50.633468 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.633431 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-credential-socket\") pod \"e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:50.633528 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.633480 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-workload-socket\") pod \"e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:50.633580 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.633518 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-istiod-ca-cert\") pod \"e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:50.633743 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.633715 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-istio-data\") pod \"e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:50.635722 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.635678 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-istio-envoy\") pod \"e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:50.635841 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.635728 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-istio-podinfo\") pod \"e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:50.641961 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.641924 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-istio-token\") pod \"e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:50.642179 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.641925 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm4nt\" (UniqueName: \"kubernetes.io/projected/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-kube-api-access-wm4nt\") pod \"e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:50.809336 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.809233 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:50.962994 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.962966 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn"] Jul 06 21:32:50.965460 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:32:50.965429 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a3b9a1_2bac_4c9e_b56d_a07e9555a4b6.slice/crio-aaf0bb805b21fb16f7a46692fa44e8cc8e093a14f9740c0f29bffe8e180471dd WatchSource:0}: Error finding container aaf0bb805b21fb16f7a46692fa44e8cc8e093a14f9740c0f29bffe8e180471dd: Status 404 returned error can't find the container with id aaf0bb805b21fb16f7a46692fa44e8cc8e093a14f9740c0f29bffe8e180471dd Jul 06 21:32:50.968111 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.968075 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:32:50.968220 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.968163 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:32:50.968220 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:50.968208 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:32:51.646565 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:51.646533 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" event={"ID":"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6","Type":"ContainerStarted","Data":"aca3809c0cce8f084ebd2b70cf9858a09fc4594a33b012d69bee54dc8522f682"} Jul 06 21:32:51.646972 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:51.646582 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" event={"ID":"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6","Type":"ContainerStarted","Data":"aaf0bb805b21fb16f7a46692fa44e8cc8e093a14f9740c0f29bffe8e180471dd"} Jul 06 21:32:51.670061 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:51.669993 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" podStartSLOduration=1.669977212 podStartE2EDuration="1.669977212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:32:51.669043069 +0000 UTC m=+2055.987266330" watchObservedRunningTime="2026-07-06 21:32:51.669977212 +0000 UTC m=+2055.988200528" Jul 06 21:32:51.809400 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:51.809358 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:52.815054 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:52.815003 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:53.657575 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:53.657543 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:53.658687 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:53.658667 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:32:54.036318 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.036224 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-778c48-b5c7c6659-cprkq"] Jul 06 21:32:54.039888 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.039867 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-778c48-b5c7c6659-cprkq" Jul 06 21:32:54.042300 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.042275 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-778c48-serving-cert\"" Jul 06 21:32:54.050120 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.050091 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-778c48-b5c7c6659-cprkq"] Jul 06 21:32:54.071363 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.071332 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztbgf\" (UniqueName: \"kubernetes.io/projected/84b5a93c-fc71-4481-a6ee-2e5f2f6e7358-kube-api-access-ztbgf\") pod \"maas-api-e2e-api-a-778c48-b5c7c6659-cprkq\" (UID: \"84b5a93c-fc71-4481-a6ee-2e5f2f6e7358\") " pod="opendatahub/maas-api-e2e-api-a-778c48-b5c7c6659-cprkq" Jul 06 21:32:54.071544 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.071390 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b5a93c-fc71-4481-a6ee-2e5f2f6e7358-openshift-service-ca\") pod \"maas-api-e2e-api-a-778c48-b5c7c6659-cprkq\" (UID: \"84b5a93c-fc71-4481-a6ee-2e5f2f6e7358\") " pod="opendatahub/maas-api-e2e-api-a-778c48-b5c7c6659-cprkq" Jul 06 21:32:54.071544 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.071506 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84b5a93c-fc71-4481-a6ee-2e5f2f6e7358-maas-api-tls\") pod \"maas-api-e2e-api-a-778c48-b5c7c6659-cprkq\" (UID: \"84b5a93c-fc71-4481-a6ee-2e5f2f6e7358\") " pod="opendatahub/maas-api-e2e-api-a-778c48-b5c7c6659-cprkq" Jul 06 21:32:54.173737 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.173700 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84b5a93c-fc71-4481-a6ee-2e5f2f6e7358-maas-api-tls\") pod \"maas-api-e2e-api-a-778c48-b5c7c6659-cprkq\" (UID: \"84b5a93c-fc71-4481-a6ee-2e5f2f6e7358\") " pod="opendatahub/maas-api-e2e-api-a-778c48-b5c7c6659-cprkq" Jul 06 21:32:54.173959 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.173793 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztbgf\" (UniqueName: \"kubernetes.io/projected/84b5a93c-fc71-4481-a6ee-2e5f2f6e7358-kube-api-access-ztbgf\") pod \"maas-api-e2e-api-a-778c48-b5c7c6659-cprkq\" (UID: \"84b5a93c-fc71-4481-a6ee-2e5f2f6e7358\") " pod="opendatahub/maas-api-e2e-api-a-778c48-b5c7c6659-cprkq" Jul 06 21:32:54.173959 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.173848 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b5a93c-fc71-4481-a6ee-2e5f2f6e7358-openshift-service-ca\") pod \"maas-api-e2e-api-a-778c48-b5c7c6659-cprkq\" (UID: \"84b5a93c-fc71-4481-a6ee-2e5f2f6e7358\") " pod="opendatahub/maas-api-e2e-api-a-778c48-b5c7c6659-cprkq" Jul 06 21:32:54.174624 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.174599 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b5a93c-fc71-4481-a6ee-2e5f2f6e7358-openshift-service-ca\") pod \"maas-api-e2e-api-a-778c48-b5c7c6659-cprkq\" (UID: \"84b5a93c-fc71-4481-a6ee-2e5f2f6e7358\") " pod="opendatahub/maas-api-e2e-api-a-778c48-b5c7c6659-cprkq" Jul 06 21:32:54.176340 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.176316 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84b5a93c-fc71-4481-a6ee-2e5f2f6e7358-maas-api-tls\") pod \"maas-api-e2e-api-a-778c48-b5c7c6659-cprkq\" (UID: \"84b5a93c-fc71-4481-a6ee-2e5f2f6e7358\") " pod="opendatahub/maas-api-e2e-api-a-778c48-b5c7c6659-cprkq" Jul 06 21:32:54.182459 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.182433 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztbgf\" (UniqueName: \"kubernetes.io/projected/84b5a93c-fc71-4481-a6ee-2e5f2f6e7358-kube-api-access-ztbgf\") pod \"maas-api-e2e-api-a-778c48-b5c7c6659-cprkq\" (UID: \"84b5a93c-fc71-4481-a6ee-2e5f2f6e7358\") " pod="opendatahub/maas-api-e2e-api-a-778c48-b5c7c6659-cprkq" Jul 06 21:32:54.354461 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.354348 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-778c48-b5c7c6659-cprkq" Jul 06 21:32:54.505773 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.505739 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-778c48-b5c7c6659-cprkq"] Jul 06 21:32:54.509025 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:32:54.508978 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b5a93c_fc71_4481_a6ee_2e5f2f6e7358.slice/crio-9827b32776df2515d3878c080c97eeb3367251d6e6886b0f3f0e170f70618759 WatchSource:0}: Error finding container 9827b32776df2515d3878c080c97eeb3367251d6e6886b0f3f0e170f70618759: Status 404 returned error can't find the container with id 9827b32776df2515d3878c080c97eeb3367251d6e6886b0f3f0e170f70618759 Jul 06 21:32:54.666175 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.666135 2577 generic.go:356] "Generic (PLEG): container finished" podID="25d4da94-ad69-4461-8572-3f50bf41515b" containerID="e9143525f043e1ce57274e48cc3c4fc85473e3643a0c51141e8af73d3433ef8b" exitCode=0 Jul 06 21:32:54.666386 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.666199 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" event={"ID":"25d4da94-ad69-4461-8572-3f50bf41515b","Type":"ContainerDied","Data":"e9143525f043e1ce57274e48cc3c4fc85473e3643a0c51141e8af73d3433ef8b"} Jul 06 21:32:54.666386 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.666266 2577 scope.go:117] "RemoveContainer" containerID="13eb914e9335308184ae5121fbe4aa8b3f501c32991ca807552dc1c94e7baa55" Jul 06 21:32:54.667811 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.667772 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-778c48-b5c7c6659-cprkq" event={"ID":"84b5a93c-fc71-4481-a6ee-2e5f2f6e7358","Type":"ContainerStarted","Data":"9827b32776df2515d3878c080c97eeb3367251d6e6886b0f3f0e170f70618759"} Jul 06 21:32:54.769530 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.769458 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:32:54.881662 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.881560 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhplr\" (UniqueName: \"kubernetes.io/projected/25d4da94-ad69-4461-8572-3f50bf41515b-kube-api-access-mhplr\") pod \"25d4da94-ad69-4461-8572-3f50bf41515b\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " Jul 06 21:32:54.881662 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.881622 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d4da94-ad69-4461-8572-3f50bf41515b-workload-certs\") pod \"25d4da94-ad69-4461-8572-3f50bf41515b\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " Jul 06 21:32:54.881898 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.881692 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d4da94-ad69-4461-8572-3f50bf41515b-istiod-ca-cert\") pod \"25d4da94-ad69-4461-8572-3f50bf41515b\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " Jul 06 21:32:54.881898 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.881731 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d4da94-ad69-4461-8572-3f50bf41515b-istio-envoy\") pod \"25d4da94-ad69-4461-8572-3f50bf41515b\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " Jul 06 21:32:54.881898 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.881800 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d4da94-ad69-4461-8572-3f50bf41515b-credential-socket\") pod \"25d4da94-ad69-4461-8572-3f50bf41515b\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " Jul 06 21:32:54.881898 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.881823 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d4da94-ad69-4461-8572-3f50bf41515b-istio-podinfo\") pod \"25d4da94-ad69-4461-8572-3f50bf41515b\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " Jul 06 21:32:54.881898 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.881851 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d4da94-ad69-4461-8572-3f50bf41515b-workload-socket\") pod \"25d4da94-ad69-4461-8572-3f50bf41515b\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " Jul 06 21:32:54.881898 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.881883 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d4da94-ad69-4461-8572-3f50bf41515b-istio-token\") pod \"25d4da94-ad69-4461-8572-3f50bf41515b\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " Jul 06 21:32:54.882317 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.881918 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d4da94-ad69-4461-8572-3f50bf41515b-istio-data\") pod \"25d4da94-ad69-4461-8572-3f50bf41515b\" (UID: \"25d4da94-ad69-4461-8572-3f50bf41515b\") " Jul 06 21:32:54.882317 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.881956 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d4da94-ad69-4461-8572-3f50bf41515b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "25d4da94-ad69-4461-8572-3f50bf41515b" (UID: "25d4da94-ad69-4461-8572-3f50bf41515b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:32:54.882317 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.882079 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d4da94-ad69-4461-8572-3f50bf41515b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "25d4da94-ad69-4461-8572-3f50bf41515b" (UID: "25d4da94-ad69-4461-8572-3f50bf41515b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:32:54.882317 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.882132 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d4da94-ad69-4461-8572-3f50bf41515b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "25d4da94-ad69-4461-8572-3f50bf41515b" (UID: "25d4da94-ad69-4461-8572-3f50bf41515b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:32:54.882317 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.882217 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d4da94-ad69-4461-8572-3f50bf41515b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "25d4da94-ad69-4461-8572-3f50bf41515b" (UID: "25d4da94-ad69-4461-8572-3f50bf41515b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:32:54.882578 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.882378 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d4da94-ad69-4461-8572-3f50bf41515b-workload-certs\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:32:54.882578 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.882396 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d4da94-ad69-4461-8572-3f50bf41515b-istiod-ca-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:32:54.882578 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.882409 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d4da94-ad69-4461-8572-3f50bf41515b-credential-socket\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:32:54.882578 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.882424 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d4da94-ad69-4461-8572-3f50bf41515b-workload-socket\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:32:54.882987 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.882959 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d4da94-ad69-4461-8572-3f50bf41515b-istio-data" (OuterVolumeSpecName: "istio-data") pod "25d4da94-ad69-4461-8572-3f50bf41515b" (UID: "25d4da94-ad69-4461-8572-3f50bf41515b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:32:54.884111 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.884085 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d4da94-ad69-4461-8572-3f50bf41515b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "25d4da94-ad69-4461-8572-3f50bf41515b" (UID: "25d4da94-ad69-4461-8572-3f50bf41515b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:32:54.884222 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.884122 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d4da94-ad69-4461-8572-3f50bf41515b-kube-api-access-mhplr" (OuterVolumeSpecName: "kube-api-access-mhplr") pod "25d4da94-ad69-4461-8572-3f50bf41515b" (UID: "25d4da94-ad69-4461-8572-3f50bf41515b"). InnerVolumeSpecName "kube-api-access-mhplr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:32:54.884364 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.884333 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25d4da94-ad69-4461-8572-3f50bf41515b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "25d4da94-ad69-4461-8572-3f50bf41515b" (UID: "25d4da94-ad69-4461-8572-3f50bf41515b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 21:32:54.884459 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.884376 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d4da94-ad69-4461-8572-3f50bf41515b-istio-token" (OuterVolumeSpecName: "istio-token") pod "25d4da94-ad69-4461-8572-3f50bf41515b" (UID: "25d4da94-ad69-4461-8572-3f50bf41515b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:32:54.983275 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.983242 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhplr\" (UniqueName: \"kubernetes.io/projected/25d4da94-ad69-4461-8572-3f50bf41515b-kube-api-access-mhplr\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:32:54.983275 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.983273 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d4da94-ad69-4461-8572-3f50bf41515b-istio-envoy\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:32:54.983467 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.983288 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d4da94-ad69-4461-8572-3f50bf41515b-istio-podinfo\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:32:54.983467 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.983299 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d4da94-ad69-4461-8572-3f50bf41515b-istio-token\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:32:54.983467 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:54.983310 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d4da94-ad69-4461-8572-3f50bf41515b-istio-data\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:32:55.674700 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:55.674656 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-778c48-b5c7c6659-cprkq" event={"ID":"84b5a93c-fc71-4481-a6ee-2e5f2f6e7358","Type":"ContainerStarted","Data":"4eb8383c74701cf38ca1a00a91a938a93d8960a02db35f5809027fdade418b4b"} Jul 06 21:32:55.675184 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:55.674733 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-778c48-b5c7c6659-cprkq" Jul 06 21:32:55.676069 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:55.676048 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" Jul 06 21:32:55.676217 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:55.676049 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj" event={"ID":"25d4da94-ad69-4461-8572-3f50bf41515b","Type":"ContainerDied","Data":"199d77e65764b05c6ab194540a502745e33c30b83718d2ae19cf752ffbcd4a41"} Jul 06 21:32:55.676217 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:55.676121 2577 scope.go:117] "RemoveContainer" containerID="e9143525f043e1ce57274e48cc3c4fc85473e3643a0c51141e8af73d3433ef8b" Jul 06 21:32:55.695211 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:55.695157 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-778c48-b5c7c6659-cprkq" podStartSLOduration=1.235662439 podStartE2EDuration="1.695137901s" totalImagesPullingTime="459.475462ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:32:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:32:55.69300301 +0000 UTC m=+2060.011226272" watchObservedRunningTime="2026-07-06 21:32:55.695137901 +0000 UTC m=+2060.013361161" Jul 06 21:32:55.709286 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:55.709251 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj"] Jul 06 21:32:55.713754 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:55.713724 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c3d033-openshift-default-c8c8b7675-d59nj"] Jul 06 21:32:56.169711 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:56.169673 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25d4da94-ad69-4461-8572-3f50bf41515b" path="/var/lib/kubelet/pods/25d4da94-ad69-4461-8572-3f50bf41515b/volumes" Jul 06 21:32:58.958282 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:58.958184 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7"] Jul 06 21:32:58.958838 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:58.958817 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25d4da94-ad69-4461-8572-3f50bf41515b" containerName="istio-proxy" Jul 06 21:32:58.958928 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:58.958841 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="25d4da94-ad69-4461-8572-3f50bf41515b" containerName="istio-proxy" Jul 06 21:32:58.958928 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:58.958858 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25d4da94-ad69-4461-8572-3f50bf41515b" containerName="istio-proxy" Jul 06 21:32:58.958928 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:58.958866 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="25d4da94-ad69-4461-8572-3f50bf41515b" containerName="istio-proxy" Jul 06 21:32:58.959124 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:58.958987 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="25d4da94-ad69-4461-8572-3f50bf41515b" containerName="istio-proxy" Jul 06 21:32:58.959124 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:58.959020 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="25d4da94-ad69-4461-8572-3f50bf41515b" containerName="istio-proxy" Jul 06 21:32:58.963932 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:58.963902 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:32:58.967606 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:58.967575 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f834a6-openshift-default-dockercfg-jxhkd\"" Jul 06 21:32:58.982056 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:58.981992 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7"] Jul 06 21:32:59.028625 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.028580 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-istio-data\") pod \"e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:32:59.028625 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.028622 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-istio-podinfo\") pod \"e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:32:59.028892 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.028685 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-workload-socket\") pod \"e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:32:59.028892 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.028760 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-credential-socket\") pod \"e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:32:59.028892 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.028803 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcwrp\" (UniqueName: \"kubernetes.io/projected/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-kube-api-access-wcwrp\") pod \"e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:32:59.029023 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.028909 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-workload-certs\") pod \"e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:32:59.029023 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.028943 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-istio-token\") pod \"e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:32:59.029023 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.028999 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-istiod-ca-cert\") pod \"e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:32:59.029151 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.029057 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-istio-envoy\") pod \"e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:32:59.130116 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.130074 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-workload-certs\") pod \"e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:32:59.130116 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.130122 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-istio-token\") pod \"e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:32:59.130386 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.130167 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-istiod-ca-cert\") pod \"e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:32:59.130386 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.130197 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-istio-envoy\") pod \"e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:32:59.130386 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.130237 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-istio-data\") pod \"e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:32:59.130386 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.130261 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-istio-podinfo\") pod \"e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:32:59.130386 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.130288 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-workload-socket\") pod \"e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:32:59.130386 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.130318 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-credential-socket\") pod \"e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:32:59.130386 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.130347 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcwrp\" (UniqueName: \"kubernetes.io/projected/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-kube-api-access-wcwrp\") pod \"e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:32:59.130767 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.130552 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-workload-certs\") pod \"e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:32:59.130835 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.130816 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-credential-socket\") pod \"e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:32:59.130898 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.130864 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-workload-socket\") pod \"e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:32:59.130952 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.130900 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-istio-data\") pod \"e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:32:59.131039 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.130988 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-istiod-ca-cert\") pod \"e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:32:59.132732 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.132705 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-istio-envoy\") pod \"e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:32:59.132990 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.132971 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-istio-podinfo\") pod \"e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:32:59.147145 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.147105 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcwrp\" (UniqueName: \"kubernetes.io/projected/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-kube-api-access-wcwrp\") pod \"e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:32:59.149140 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.149117 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-istio-token\") pod \"e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:32:59.275376 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.275272 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:32:59.463873 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.463830 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7"] Jul 06 21:32:59.473561 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:32:59.473531 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d734cf1_9c31_4bf3_bed8_2c473389fa4b.slice/crio-c478b2a6aec08e15790290435ce0cb5f567566456732aca55850e5262eccae94 WatchSource:0}: Error finding container c478b2a6aec08e15790290435ce0cb5f567566456732aca55850e5262eccae94: Status 404 returned error can't find the container with id c478b2a6aec08e15790290435ce0cb5f567566456732aca55850e5262eccae94 Jul 06 21:32:59.475809 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.475773 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:32:59.475916 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.475859 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:32:59.475916 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.475902 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:32:59.702369 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.702317 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" event={"ID":"3d734cf1-9c31-4bf3-bed8-2c473389fa4b","Type":"ContainerStarted","Data":"273ad7809168f7ea27e85fb7f358becb99e6f6cfc0be8a4db3fd46bc619874c2"} Jul 06 21:32:59.702582 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.702405 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" event={"ID":"3d734cf1-9c31-4bf3-bed8-2c473389fa4b","Type":"ContainerStarted","Data":"c478b2a6aec08e15790290435ce0cb5f567566456732aca55850e5262eccae94"} Jul 06 21:32:59.728763 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:32:59.728698 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" podStartSLOduration=1.728677715 podStartE2EDuration="1.728677715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:32:59.725695068 +0000 UTC m=+2064.043918330" watchObservedRunningTime="2026-07-06 21:32:59.728677715 +0000 UTC m=+2064.046900977" Jul 06 21:33:00.275642 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:00.275597 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:33:00.280871 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:00.280841 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:33:00.707568 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:00.707536 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:33:00.708557 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:00.708534 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:33:01.689966 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:01.689936 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-778c48-b5c7c6659-cprkq" Jul 06 21:33:09.023093 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:09.022902 2577 patch_prober.go:28] interesting pod/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": read tcp 10.133.0.2:47556->10.133.0.60:15021: read: connection reset by peer" start-of-body= Jul 06 21:33:09.023093 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:09.022964 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" podUID="d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": read tcp 10.133.0.2:47556->10.133.0.60:15021: read: connection reset by peer" Jul 06 21:33:09.763789 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:09.763756 2577 generic.go:356] "Generic (PLEG): container finished" podID="d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6" containerID="aca3809c0cce8f084ebd2b70cf9858a09fc4594a33b012d69bee54dc8522f682" exitCode=0 Jul 06 21:33:09.763981 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:09.763830 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" event={"ID":"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6","Type":"ContainerDied","Data":"aca3809c0cce8f084ebd2b70cf9858a09fc4594a33b012d69bee54dc8522f682"} Jul 06 21:33:09.764304 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:09.764284 2577 scope.go:117] "RemoveContainer" containerID="aca3809c0cce8f084ebd2b70cf9858a09fc4594a33b012d69bee54dc8522f682" Jul 06 21:33:09.765378 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:09.765350 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:33:09.765445 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:09.765425 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:33:09.765487 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:09.765462 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:33:10.771043 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:10.770980 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" event={"ID":"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6","Type":"ContainerStarted","Data":"7a9a528eeaaf8835ea93fd913a6e7d8849bc3e5985b70ee9317d6f7829f72034"} Jul 06 21:33:10.809845 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:10.809802 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:33:10.810181 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:10.810154 2577 patch_prober.go:28] interesting pod/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 06 21:33:10.810254 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:10.810210 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" podUID="d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 06 21:33:11.810495 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:11.810453 2577 patch_prober.go:28] interesting pod/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 06 21:33:11.810904 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:11.810510 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" podUID="d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 06 21:33:12.810715 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:12.810678 2577 patch_prober.go:28] interesting pod/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 06 21:33:12.811177 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:12.810748 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" podUID="d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 06 21:33:13.810455 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:13.810420 2577 patch_prober.go:28] interesting pod/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 06 21:33:13.810654 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:13.810484 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" podUID="d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 06 21:33:14.810689 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:14.810649 2577 patch_prober.go:28] interesting pod/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 06 21:33:14.811104 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:14.810725 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" podUID="d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 06 21:33:15.242035 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:15.241975 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-778c48-b5c7c6659-cprkq"] Jul 06 21:33:15.243085 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:15.242250 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-778c48-b5c7c6659-cprkq" podUID="84b5a93c-fc71-4481-a6ee-2e5f2f6e7358" containerName="maas-api" containerID="cri-o://4eb8383c74701cf38ca1a00a91a938a93d8960a02db35f5809027fdade418b4b" gracePeriod=30 Jul 06 21:33:15.510768 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:15.510685 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-778c48-b5c7c6659-cprkq" Jul 06 21:33:15.621034 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:15.620971 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84b5a93c-fc71-4481-a6ee-2e5f2f6e7358-maas-api-tls\") pod \"84b5a93c-fc71-4481-a6ee-2e5f2f6e7358\" (UID: \"84b5a93c-fc71-4481-a6ee-2e5f2f6e7358\") " Jul 06 21:33:15.621254 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:15.621048 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztbgf\" (UniqueName: \"kubernetes.io/projected/84b5a93c-fc71-4481-a6ee-2e5f2f6e7358-kube-api-access-ztbgf\") pod \"84b5a93c-fc71-4481-a6ee-2e5f2f6e7358\" (UID: \"84b5a93c-fc71-4481-a6ee-2e5f2f6e7358\") " Jul 06 21:33:15.621254 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:15.621161 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b5a93c-fc71-4481-a6ee-2e5f2f6e7358-openshift-service-ca\") pod \"84b5a93c-fc71-4481-a6ee-2e5f2f6e7358\" (UID: \"84b5a93c-fc71-4481-a6ee-2e5f2f6e7358\") " Jul 06 21:33:15.621591 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:15.621561 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b5a93c-fc71-4481-a6ee-2e5f2f6e7358-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "84b5a93c-fc71-4481-a6ee-2e5f2f6e7358" (UID: "84b5a93c-fc71-4481-a6ee-2e5f2f6e7358"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:33:15.623355 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:15.623329 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84b5a93c-fc71-4481-a6ee-2e5f2f6e7358-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "84b5a93c-fc71-4481-a6ee-2e5f2f6e7358" (UID: "84b5a93c-fc71-4481-a6ee-2e5f2f6e7358"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:33:15.623355 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:15.623345 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b5a93c-fc71-4481-a6ee-2e5f2f6e7358-kube-api-access-ztbgf" (OuterVolumeSpecName: "kube-api-access-ztbgf") pod "84b5a93c-fc71-4481-a6ee-2e5f2f6e7358" (UID: "84b5a93c-fc71-4481-a6ee-2e5f2f6e7358"). InnerVolumeSpecName "kube-api-access-ztbgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:33:15.722153 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:15.722107 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b5a93c-fc71-4481-a6ee-2e5f2f6e7358-openshift-service-ca\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:33:15.722153 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:15.722145 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84b5a93c-fc71-4481-a6ee-2e5f2f6e7358-maas-api-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:33:15.722153 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:15.722159 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztbgf\" (UniqueName: \"kubernetes.io/projected/84b5a93c-fc71-4481-a6ee-2e5f2f6e7358-kube-api-access-ztbgf\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:33:15.802392 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:15.802300 2577 generic.go:356] "Generic (PLEG): container finished" podID="84b5a93c-fc71-4481-a6ee-2e5f2f6e7358" containerID="4eb8383c74701cf38ca1a00a91a938a93d8960a02db35f5809027fdade418b4b" exitCode=0 Jul 06 21:33:15.802392 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:15.802349 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-778c48-b5c7c6659-cprkq" Jul 06 21:33:15.802580 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:15.802387 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-778c48-b5c7c6659-cprkq" event={"ID":"84b5a93c-fc71-4481-a6ee-2e5f2f6e7358","Type":"ContainerDied","Data":"4eb8383c74701cf38ca1a00a91a938a93d8960a02db35f5809027fdade418b4b"} Jul 06 21:33:15.802580 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:15.802451 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-778c48-b5c7c6659-cprkq" event={"ID":"84b5a93c-fc71-4481-a6ee-2e5f2f6e7358","Type":"ContainerDied","Data":"9827b32776df2515d3878c080c97eeb3367251d6e6886b0f3f0e170f70618759"} Jul 06 21:33:15.802580 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:15.802480 2577 scope.go:117] "RemoveContainer" containerID="4eb8383c74701cf38ca1a00a91a938a93d8960a02db35f5809027fdade418b4b" Jul 06 21:33:15.814095 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:15.814072 2577 scope.go:117] "RemoveContainer" containerID="4eb8383c74701cf38ca1a00a91a938a93d8960a02db35f5809027fdade418b4b" Jul 06 21:33:15.814486 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:15.814071 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:33:15.814572 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:33:15.814520 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eb8383c74701cf38ca1a00a91a938a93d8960a02db35f5809027fdade418b4b\": container with ID starting with 4eb8383c74701cf38ca1a00a91a938a93d8960a02db35f5809027fdade418b4b not found: ID does not exist" containerID="4eb8383c74701cf38ca1a00a91a938a93d8960a02db35f5809027fdade418b4b" Jul 06 21:33:15.814572 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:15.814551 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:33:15.814572 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:15.814557 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eb8383c74701cf38ca1a00a91a938a93d8960a02db35f5809027fdade418b4b"} err="failed to get container status \"4eb8383c74701cf38ca1a00a91a938a93d8960a02db35f5809027fdade418b4b\": rpc error: code = NotFound desc = could not find container \"4eb8383c74701cf38ca1a00a91a938a93d8960a02db35f5809027fdade418b4b\": container with ID starting with 4eb8383c74701cf38ca1a00a91a938a93d8960a02db35f5809027fdade418b4b not found: ID does not exist" Jul 06 21:33:15.815419 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:15.815399 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:33:15.827080 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:15.827047 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-778c48-b5c7c6659-cprkq"] Jul 06 21:33:15.830891 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:15.830856 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-778c48-b5c7c6659-cprkq"] Jul 06 21:33:16.170487 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:16.170440 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84b5a93c-fc71-4481-a6ee-2e5f2f6e7358" path="/var/lib/kubelet/pods/84b5a93c-fc71-4481-a6ee-2e5f2f6e7358/volumes" Jul 06 21:33:16.750264 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:16.750218 2577 patch_prober.go:28] interesting pod/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 06 21:33:16.750455 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:16.750300 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" podUID="3d734cf1-9c31-4bf3-bed8-2c473389fa4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 21:33:18.819731 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:18.819696 2577 generic.go:356] "Generic (PLEG): container finished" podID="3d734cf1-9c31-4bf3-bed8-2c473389fa4b" containerID="273ad7809168f7ea27e85fb7f358becb99e6f6cfc0be8a4db3fd46bc619874c2" exitCode=0 Jul 06 21:33:18.820164 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:18.819765 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" event={"ID":"3d734cf1-9c31-4bf3-bed8-2c473389fa4b","Type":"ContainerDied","Data":"273ad7809168f7ea27e85fb7f358becb99e6f6cfc0be8a4db3fd46bc619874c2"} Jul 06 21:33:18.820207 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:18.820185 2577 scope.go:117] "RemoveContainer" containerID="273ad7809168f7ea27e85fb7f358becb99e6f6cfc0be8a4db3fd46bc619874c2" Jul 06 21:33:18.821304 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:18.821275 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:33:18.821384 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:18.821357 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:33:18.821427 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:18.821391 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:33:19.827633 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:19.827589 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" event={"ID":"3d734cf1-9c31-4bf3-bed8-2c473389fa4b","Type":"ContainerStarted","Data":"def10810689097730ee63f0c9fec4e2315e4c855e5d852fbcebcbf9a1b5ae297"} Jul 06 21:33:20.275723 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:20.275687 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:33:20.275961 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:20.275942 2577 patch_prober.go:28] interesting pod/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 06 21:33:20.276028 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:20.275986 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" podUID="3d734cf1-9c31-4bf3-bed8-2c473389fa4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 06 21:33:21.276761 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:21.276725 2577 patch_prober.go:28] interesting pod/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 06 21:33:21.277365 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:21.277314 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" podUID="3d734cf1-9c31-4bf3-bed8-2c473389fa4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 06 21:33:21.300954 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:21.300911 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn"] Jul 06 21:33:21.301254 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:21.301220 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" podUID="d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6" containerName="istio-proxy" containerID="cri-o://7a9a528eeaaf8835ea93fd913a6e7d8849bc3e5985b70ee9317d6f7829f72034" gracePeriod=30 Jul 06 21:33:22.275841 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:22.275805 2577 patch_prober.go:28] interesting pod/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 06 21:33:22.276067 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:22.275860 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" podUID="3d734cf1-9c31-4bf3-bed8-2c473389fa4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 06 21:33:23.276603 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:23.276563 2577 patch_prober.go:28] interesting pod/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 06 21:33:23.277037 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:23.276623 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" podUID="3d734cf1-9c31-4bf3-bed8-2c473389fa4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 06 21:33:24.279616 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:24.279583 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:33:24.280045 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:24.279841 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:33:24.280539 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:24.280521 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:33:26.559779 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.559752 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:33:26.625989 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.625954 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-istio-podinfo\") pod \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " Jul 06 21:33:26.626174 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.626001 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-istio-data\") pod \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " Jul 06 21:33:26.626174 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.626035 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-workload-certs\") pod \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " Jul 06 21:33:26.626246 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.626224 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-istio-envoy\") pod \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " Jul 06 21:33:26.626295 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.626280 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-istio-token\") pod \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " Jul 06 21:33:26.626393 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.626376 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-workload-socket\") pod \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " Jul 06 21:33:26.626457 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.626414 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-credential-socket\") pod \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " Jul 06 21:33:26.626517 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.626278 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6" (UID: "d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:33:26.626572 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.626535 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wm4nt\" (UniqueName: \"kubernetes.io/projected/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-kube-api-access-wm4nt\") pod \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " Jul 06 21:33:26.626631 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.626572 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-istiod-ca-cert\") pod \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\" (UID: \"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6\") " Jul 06 21:33:26.626688 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.626623 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6" (UID: "d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:33:26.626688 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.626657 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6" (UID: "d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:33:26.626940 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.626915 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-workload-socket\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:33:26.626940 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.626941 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-credential-socket\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:33:26.627116 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.626954 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-workload-certs\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:33:26.627116 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.626986 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6" (UID: "d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:33:26.627116 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.627049 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-istio-data" (OuterVolumeSpecName: "istio-data") pod "d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6" (UID: "d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:33:26.628490 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.628458 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-istio-token" (OuterVolumeSpecName: "istio-token") pod "d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6" (UID: "d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:33:26.628614 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.628598 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-kube-api-access-wm4nt" (OuterVolumeSpecName: "kube-api-access-wm4nt") pod "d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6" (UID: "d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6"). InnerVolumeSpecName "kube-api-access-wm4nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:33:26.628733 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.628712 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6" (UID: "d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 21:33:26.628733 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.628721 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6" (UID: "d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:33:26.727526 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.727489 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-istiod-ca-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:33:26.727526 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.727522 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-istio-podinfo\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:33:26.727526 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.727535 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-istio-data\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:33:26.727787 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.727544 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-istio-envoy\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:33:26.727787 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.727552 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-istio-token\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:33:26.727787 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.727560 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm4nt\" (UniqueName: \"kubernetes.io/projected/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6-kube-api-access-wm4nt\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:33:26.874118 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.874082 2577 generic.go:356] "Generic (PLEG): container finished" podID="d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6" containerID="7a9a528eeaaf8835ea93fd913a6e7d8849bc3e5985b70ee9317d6f7829f72034" exitCode=0 Jul 06 21:33:26.874309 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.874161 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" Jul 06 21:33:26.874309 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.874175 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" event={"ID":"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6","Type":"ContainerDied","Data":"7a9a528eeaaf8835ea93fd913a6e7d8849bc3e5985b70ee9317d6f7829f72034"} Jul 06 21:33:26.874309 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.874259 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn" event={"ID":"d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6","Type":"ContainerDied","Data":"aaf0bb805b21fb16f7a46692fa44e8cc8e093a14f9740c0f29bffe8e180471dd"} Jul 06 21:33:26.874309 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.874284 2577 scope.go:117] "RemoveContainer" containerID="7a9a528eeaaf8835ea93fd913a6e7d8849bc3e5985b70ee9317d6f7829f72034" Jul 06 21:33:26.884211 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.884187 2577 scope.go:117] "RemoveContainer" containerID="aca3809c0cce8f084ebd2b70cf9858a09fc4594a33b012d69bee54dc8522f682" Jul 06 21:33:26.893259 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.893237 2577 scope.go:117] "RemoveContainer" containerID="7a9a528eeaaf8835ea93fd913a6e7d8849bc3e5985b70ee9317d6f7829f72034" Jul 06 21:33:26.893597 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:33:26.893578 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a9a528eeaaf8835ea93fd913a6e7d8849bc3e5985b70ee9317d6f7829f72034\": container with ID starting with 7a9a528eeaaf8835ea93fd913a6e7d8849bc3e5985b70ee9317d6f7829f72034 not found: ID does not exist" containerID="7a9a528eeaaf8835ea93fd913a6e7d8849bc3e5985b70ee9317d6f7829f72034" Jul 06 21:33:26.893644 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.893607 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a9a528eeaaf8835ea93fd913a6e7d8849bc3e5985b70ee9317d6f7829f72034"} err="failed to get container status \"7a9a528eeaaf8835ea93fd913a6e7d8849bc3e5985b70ee9317d6f7829f72034\": rpc error: code = NotFound desc = could not find container \"7a9a528eeaaf8835ea93fd913a6e7d8849bc3e5985b70ee9317d6f7829f72034\": container with ID starting with 7a9a528eeaaf8835ea93fd913a6e7d8849bc3e5985b70ee9317d6f7829f72034 not found: ID does not exist" Jul 06 21:33:26.893644 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.893628 2577 scope.go:117] "RemoveContainer" containerID="aca3809c0cce8f084ebd2b70cf9858a09fc4594a33b012d69bee54dc8522f682" Jul 06 21:33:26.893878 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:33:26.893860 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aca3809c0cce8f084ebd2b70cf9858a09fc4594a33b012d69bee54dc8522f682\": container with ID starting with aca3809c0cce8f084ebd2b70cf9858a09fc4594a33b012d69bee54dc8522f682 not found: ID does not exist" containerID="aca3809c0cce8f084ebd2b70cf9858a09fc4594a33b012d69bee54dc8522f682" Jul 06 21:33:26.893932 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.893884 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aca3809c0cce8f084ebd2b70cf9858a09fc4594a33b012d69bee54dc8522f682"} err="failed to get container status \"aca3809c0cce8f084ebd2b70cf9858a09fc4594a33b012d69bee54dc8522f682\": rpc error: code = NotFound desc = could not find container \"aca3809c0cce8f084ebd2b70cf9858a09fc4594a33b012d69bee54dc8522f682\": container with ID starting with aca3809c0cce8f084ebd2b70cf9858a09fc4594a33b012d69bee54dc8522f682 not found: ID does not exist" Jul 06 21:33:26.897781 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.897744 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn"] Jul 06 21:33:26.901989 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:26.901959 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-778c48-openshift-default-cfd87c6c7-7dgxn"] Jul 06 21:33:28.169380 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:28.169342 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6" path="/var/lib/kubelet/pods/d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6/volumes" Jul 06 21:33:28.544061 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:28.543963 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7"] Jul 06 21:33:28.544242 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:28.544210 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" podUID="3d734cf1-9c31-4bf3-bed8-2c473389fa4b" containerName="istio-proxy" containerID="cri-o://def10810689097730ee63f0c9fec4e2315e4c855e5d852fbcebcbf9a1b5ae297" gracePeriod=30 Jul 06 21:33:29.425811 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.425775 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b"] Jul 06 21:33:29.426396 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.426197 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6" containerName="istio-proxy" Jul 06 21:33:29.426396 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.426209 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6" containerName="istio-proxy" Jul 06 21:33:29.426396 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.426232 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84b5a93c-fc71-4481-a6ee-2e5f2f6e7358" containerName="maas-api" Jul 06 21:33:29.426396 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.426239 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="84b5a93c-fc71-4481-a6ee-2e5f2f6e7358" containerName="maas-api" Jul 06 21:33:29.426396 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.426298 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6" containerName="istio-proxy" Jul 06 21:33:29.426396 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.426308 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="84b5a93c-fc71-4481-a6ee-2e5f2f6e7358" containerName="maas-api" Jul 06 21:33:29.429444 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.429417 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:29.432082 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.432059 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-900ebe-openshift-default-dockercfg-wttvw\"" Jul 06 21:33:29.443195 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.443164 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b"] Jul 06 21:33:29.560387 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.560344 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/490125d5-40f7-468d-85f5-41d91b91bfc2-istio-envoy\") pod \"e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:29.560578 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.560400 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/490125d5-40f7-468d-85f5-41d91b91bfc2-istio-podinfo\") pod \"e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:29.560578 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.560457 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/490125d5-40f7-468d-85f5-41d91b91bfc2-credential-socket\") pod \"e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:29.560578 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.560546 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/490125d5-40f7-468d-85f5-41d91b91bfc2-workload-certs\") pod \"e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:29.560683 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.560615 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/490125d5-40f7-468d-85f5-41d91b91bfc2-istiod-ca-cert\") pod \"e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:29.560683 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.560640 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/490125d5-40f7-468d-85f5-41d91b91bfc2-workload-socket\") pod \"e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:29.560751 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.560696 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/490125d5-40f7-468d-85f5-41d91b91bfc2-istio-data\") pod \"e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:29.560751 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.560720 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwwpl\" (UniqueName: \"kubernetes.io/projected/490125d5-40f7-468d-85f5-41d91b91bfc2-kube-api-access-mwwpl\") pod \"e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:29.560751 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.560741 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/490125d5-40f7-468d-85f5-41d91b91bfc2-istio-token\") pod \"e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:29.661779 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.661737 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/490125d5-40f7-468d-85f5-41d91b91bfc2-istiod-ca-cert\") pod \"e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:29.661779 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.661779 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/490125d5-40f7-468d-85f5-41d91b91bfc2-workload-socket\") pod \"e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:29.662108 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.661902 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/490125d5-40f7-468d-85f5-41d91b91bfc2-istio-data\") pod \"e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:29.662108 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.661922 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwwpl\" (UniqueName: \"kubernetes.io/projected/490125d5-40f7-468d-85f5-41d91b91bfc2-kube-api-access-mwwpl\") pod \"e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:29.662108 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.661940 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/490125d5-40f7-468d-85f5-41d91b91bfc2-istio-token\") pod \"e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:29.662108 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.661964 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/490125d5-40f7-468d-85f5-41d91b91bfc2-istio-envoy\") pod \"e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:29.662329 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.662207 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/490125d5-40f7-468d-85f5-41d91b91bfc2-istio-podinfo\") pod \"e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:29.662329 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.662255 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/490125d5-40f7-468d-85f5-41d91b91bfc2-credential-socket\") pod \"e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:29.662447 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.662340 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/490125d5-40f7-468d-85f5-41d91b91bfc2-workload-certs\") pod \"e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:29.662447 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.662384 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/490125d5-40f7-468d-85f5-41d91b91bfc2-istio-data\") pod \"e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:29.662447 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.662335 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/490125d5-40f7-468d-85f5-41d91b91bfc2-workload-socket\") pod \"e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:29.662615 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.662581 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/490125d5-40f7-468d-85f5-41d91b91bfc2-workload-certs\") pod \"e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:29.662674 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.662620 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/490125d5-40f7-468d-85f5-41d91b91bfc2-credential-socket\") pod \"e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:29.662729 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.662668 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/490125d5-40f7-468d-85f5-41d91b91bfc2-istiod-ca-cert\") pod \"e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:29.664593 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.664572 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/490125d5-40f7-468d-85f5-41d91b91bfc2-istio-envoy\") pod \"e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:29.664910 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.664885 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/490125d5-40f7-468d-85f5-41d91b91bfc2-istio-podinfo\") pod \"e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:29.674423 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.674385 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/490125d5-40f7-468d-85f5-41d91b91bfc2-istio-token\") pod \"e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:29.674548 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.674490 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwwpl\" (UniqueName: \"kubernetes.io/projected/490125d5-40f7-468d-85f5-41d91b91bfc2-kube-api-access-mwwpl\") pod \"e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:29.743579 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.743479 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:29.885657 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.885626 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b"] Jul 06 21:33:29.887381 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:33:29.887353 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod490125d5_40f7_468d_85f5_41d91b91bfc2.slice/crio-c51236bc1571e9a57f48ac9051deab671fe9853235da7af6bd68ae55ea6c83ae WatchSource:0}: Error finding container c51236bc1571e9a57f48ac9051deab671fe9853235da7af6bd68ae55ea6c83ae: Status 404 returned error can't find the container with id c51236bc1571e9a57f48ac9051deab671fe9853235da7af6bd68ae55ea6c83ae Jul 06 21:33:29.889739 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.889696 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:33:29.889870 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.889770 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:33:29.889870 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.889802 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:33:29.894926 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:29.894901 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" event={"ID":"490125d5-40f7-468d-85f5-41d91b91bfc2","Type":"ContainerStarted","Data":"c51236bc1571e9a57f48ac9051deab671fe9853235da7af6bd68ae55ea6c83ae"} Jul 06 21:33:30.902143 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:30.902103 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" event={"ID":"490125d5-40f7-468d-85f5-41d91b91bfc2","Type":"ContainerStarted","Data":"c422ce290a423ca5088de9b5a12b9cff5786080b957fb9934bcfcc9fc169e24c"} Jul 06 21:33:30.925782 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:30.925728 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" podStartSLOduration=1.925711972 podStartE2EDuration="1.925711972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:33:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:33:30.923566232 +0000 UTC m=+2095.241789515" watchObservedRunningTime="2026-07-06 21:33:30.925711972 +0000 UTC m=+2095.243935232" Jul 06 21:33:31.743712 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:31.743665 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:31.748996 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:31.748962 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:31.908602 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:31.908562 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:31.909880 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:31.909856 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:33.796981 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.796951 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:33:33.903133 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.903099 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-workload-certs\") pod \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " Jul 06 21:33:33.903344 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.903162 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-istiod-ca-cert\") pod \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " Jul 06 21:33:33.903344 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.903215 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-istio-podinfo\") pod \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " Jul 06 21:33:33.903344 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.903262 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-istio-envoy\") pod \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " Jul 06 21:33:33.903344 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.903286 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-credential-socket\") pod \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " Jul 06 21:33:33.903344 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.903311 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-istio-token\") pod \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " Jul 06 21:33:33.903344 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.903337 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-istio-data\") pod \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " Jul 06 21:33:33.903643 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.903366 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-workload-socket\") pod \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " Jul 06 21:33:33.903643 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.903433 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3d734cf1-9c31-4bf3-bed8-2c473389fa4b" (UID: "3d734cf1-9c31-4bf3-bed8-2c473389fa4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:33:33.903643 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.903485 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcwrp\" (UniqueName: \"kubernetes.io/projected/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-kube-api-access-wcwrp\") pod \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\" (UID: \"3d734cf1-9c31-4bf3-bed8-2c473389fa4b\") " Jul 06 21:33:33.903643 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.903544 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3d734cf1-9c31-4bf3-bed8-2c473389fa4b" (UID: "3d734cf1-9c31-4bf3-bed8-2c473389fa4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:33:33.903824 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.903673 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3d734cf1-9c31-4bf3-bed8-2c473389fa4b" (UID: "3d734cf1-9c31-4bf3-bed8-2c473389fa4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:33:33.903824 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.903597 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3d734cf1-9c31-4bf3-bed8-2c473389fa4b" (UID: "3d734cf1-9c31-4bf3-bed8-2c473389fa4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:33:33.903937 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.903846 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-workload-socket\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:33:33.903937 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.903864 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-workload-certs\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:33:33.903937 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.903876 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-istiod-ca-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:33:33.903937 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.903888 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-credential-socket\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:33:33.904438 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.904405 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-istio-data" (OuterVolumeSpecName: "istio-data") pod "3d734cf1-9c31-4bf3-bed8-2c473389fa4b" (UID: "3d734cf1-9c31-4bf3-bed8-2c473389fa4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:33:33.905693 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.905666 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-istio-token" (OuterVolumeSpecName: "istio-token") pod "3d734cf1-9c31-4bf3-bed8-2c473389fa4b" (UID: "3d734cf1-9c31-4bf3-bed8-2c473389fa4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:33:33.905810 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.905706 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3d734cf1-9c31-4bf3-bed8-2c473389fa4b" (UID: "3d734cf1-9c31-4bf3-bed8-2c473389fa4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:33:33.906097 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.906075 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-kube-api-access-wcwrp" (OuterVolumeSpecName: "kube-api-access-wcwrp") pod "3d734cf1-9c31-4bf3-bed8-2c473389fa4b" (UID: "3d734cf1-9c31-4bf3-bed8-2c473389fa4b"). InnerVolumeSpecName "kube-api-access-wcwrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:33:33.906176 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.906106 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3d734cf1-9c31-4bf3-bed8-2c473389fa4b" (UID: "3d734cf1-9c31-4bf3-bed8-2c473389fa4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 21:33:33.920625 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.920592 2577 generic.go:356] "Generic (PLEG): container finished" podID="3d734cf1-9c31-4bf3-bed8-2c473389fa4b" containerID="def10810689097730ee63f0c9fec4e2315e4c855e5d852fbcebcbf9a1b5ae297" exitCode=0 Jul 06 21:33:33.920804 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.920647 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" Jul 06 21:33:33.920804 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.920671 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" event={"ID":"3d734cf1-9c31-4bf3-bed8-2c473389fa4b","Type":"ContainerDied","Data":"def10810689097730ee63f0c9fec4e2315e4c855e5d852fbcebcbf9a1b5ae297"} Jul 06 21:33:33.920804 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.920727 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7" event={"ID":"3d734cf1-9c31-4bf3-bed8-2c473389fa4b","Type":"ContainerDied","Data":"c478b2a6aec08e15790290435ce0cb5f567566456732aca55850e5262eccae94"} Jul 06 21:33:33.920804 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.920749 2577 scope.go:117] "RemoveContainer" containerID="def10810689097730ee63f0c9fec4e2315e4c855e5d852fbcebcbf9a1b5ae297" Jul 06 21:33:33.930740 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.930721 2577 scope.go:117] "RemoveContainer" containerID="273ad7809168f7ea27e85fb7f358becb99e6f6cfc0be8a4db3fd46bc619874c2" Jul 06 21:33:33.939450 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.939428 2577 scope.go:117] "RemoveContainer" containerID="def10810689097730ee63f0c9fec4e2315e4c855e5d852fbcebcbf9a1b5ae297" Jul 06 21:33:33.939747 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:33:33.939726 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"def10810689097730ee63f0c9fec4e2315e4c855e5d852fbcebcbf9a1b5ae297\": container with ID starting with def10810689097730ee63f0c9fec4e2315e4c855e5d852fbcebcbf9a1b5ae297 not found: ID does not exist" containerID="def10810689097730ee63f0c9fec4e2315e4c855e5d852fbcebcbf9a1b5ae297" Jul 06 21:33:33.939818 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.939757 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"def10810689097730ee63f0c9fec4e2315e4c855e5d852fbcebcbf9a1b5ae297"} err="failed to get container status \"def10810689097730ee63f0c9fec4e2315e4c855e5d852fbcebcbf9a1b5ae297\": rpc error: code = NotFound desc = could not find container \"def10810689097730ee63f0c9fec4e2315e4c855e5d852fbcebcbf9a1b5ae297\": container with ID starting with def10810689097730ee63f0c9fec4e2315e4c855e5d852fbcebcbf9a1b5ae297 not found: ID does not exist" Jul 06 21:33:33.939818 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.939776 2577 scope.go:117] "RemoveContainer" containerID="273ad7809168f7ea27e85fb7f358becb99e6f6cfc0be8a4db3fd46bc619874c2" Jul 06 21:33:33.940075 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:33:33.940049 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"273ad7809168f7ea27e85fb7f358becb99e6f6cfc0be8a4db3fd46bc619874c2\": container with ID starting with 273ad7809168f7ea27e85fb7f358becb99e6f6cfc0be8a4db3fd46bc619874c2 not found: ID does not exist" containerID="273ad7809168f7ea27e85fb7f358becb99e6f6cfc0be8a4db3fd46bc619874c2" Jul 06 21:33:33.940131 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.940081 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273ad7809168f7ea27e85fb7f358becb99e6f6cfc0be8a4db3fd46bc619874c2"} err="failed to get container status \"273ad7809168f7ea27e85fb7f358becb99e6f6cfc0be8a4db3fd46bc619874c2\": rpc error: code = NotFound desc = could not find container \"273ad7809168f7ea27e85fb7f358becb99e6f6cfc0be8a4db3fd46bc619874c2\": container with ID starting with 273ad7809168f7ea27e85fb7f358becb99e6f6cfc0be8a4db3fd46bc619874c2 not found: ID does not exist" Jul 06 21:33:33.943452 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.943424 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7"] Jul 06 21:33:33.948135 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:33.948108 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f834a6-openshift-default-58d8759c77-vmcz7"] Jul 06 21:33:34.005348 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:34.005312 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcwrp\" (UniqueName: \"kubernetes.io/projected/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-kube-api-access-wcwrp\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:33:34.005348 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:34.005347 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-istio-podinfo\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:33:34.005610 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:34.005365 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-istio-envoy\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:33:34.005610 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:34.005377 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-istio-token\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:33:34.005610 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:34.005390 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d734cf1-9c31-4bf3-bed8-2c473389fa4b-istio-data\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:33:34.169781 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:34.169693 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d734cf1-9c31-4bf3-bed8-2c473389fa4b" path="/var/lib/kubelet/pods/3d734cf1-9c31-4bf3-bed8-2c473389fa4b/volumes" Jul 06 21:33:36.331732 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:36.331705 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qjs2n_64b45669-ed56-41b9-a3ed-1fe5c13508b6/ovn-acl-logging/0.log" Jul 06 21:33:36.335588 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:36.335566 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qjs2n_64b45669-ed56-41b9-a3ed-1fe5c13508b6/ovn-acl-logging/0.log" Jul 06 21:33:38.009665 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.009615 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s"] Jul 06 21:33:38.010254 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.010237 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d734cf1-9c31-4bf3-bed8-2c473389fa4b" containerName="istio-proxy" Jul 06 21:33:38.010342 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.010256 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d734cf1-9c31-4bf3-bed8-2c473389fa4b" containerName="istio-proxy" Jul 06 21:33:38.010342 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.010270 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d734cf1-9c31-4bf3-bed8-2c473389fa4b" containerName="istio-proxy" Jul 06 21:33:38.010342 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.010280 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d734cf1-9c31-4bf3-bed8-2c473389fa4b" containerName="istio-proxy" Jul 06 21:33:38.010342 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.010316 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6" containerName="istio-proxy" Jul 06 21:33:38.010342 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.010325 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6" containerName="istio-proxy" Jul 06 21:33:38.010633 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.010429 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d734cf1-9c31-4bf3-bed8-2c473389fa4b" containerName="istio-proxy" Jul 06 21:33:38.010633 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.010440 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7a3b9a1-2bac-4c9e-b56d-a07e9555a4b6" containerName="istio-proxy" Jul 06 21:33:38.014483 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.014455 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:38.017207 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.017182 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-5ea078-openshift-default-dockercfg-cm2p6\"" Jul 06 21:33:38.024666 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.024635 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s"] Jul 06 21:33:38.150563 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.150520 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-istio-envoy\") pod \"e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:38.150929 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.150899 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mggx\" (UniqueName: \"kubernetes.io/projected/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-kube-api-access-5mggx\") pod \"e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:38.151203 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.151181 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-credential-socket\") pod \"e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:38.151371 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.151353 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-istio-token\") pod \"e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:38.151487 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.151472 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-istio-podinfo\") pod \"e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:38.151605 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.151589 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-istiod-ca-cert\") pod \"e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:38.151730 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.151715 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-workload-socket\") pod \"e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:38.151853 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.151837 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-workload-certs\") pod \"e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:38.151973 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.151954 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-istio-data\") pod \"e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:38.253594 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.253346 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-workload-socket\") pod \"e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:38.253941 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.253803 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-workload-socket\") pod \"e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:38.254115 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.253878 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-workload-certs\") pod \"e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:38.254266 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.254244 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-istio-data\") pod \"e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:38.254411 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.254294 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-istio-envoy\") pod \"e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:38.254411 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.254325 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-workload-certs\") pod \"e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:38.254411 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.254347 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mggx\" (UniqueName: \"kubernetes.io/projected/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-kube-api-access-5mggx\") pod \"e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:38.254601 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.254494 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-credential-socket\") pod \"e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:38.254601 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.254553 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-istio-token\") pod \"e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:38.254601 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.254595 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-istio-podinfo\") pod \"e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:38.254773 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.254628 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-istiod-ca-cert\") pod \"e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:38.254773 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.254709 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-istio-data\") pod \"e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:38.255420 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.255394 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-istiod-ca-cert\") pod \"e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:38.255561 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.255536 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-credential-socket\") pod \"e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:38.257612 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.257578 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-istio-envoy\") pod \"e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:38.257992 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.257970 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-istio-podinfo\") pod \"e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:38.263112 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.262949 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-istio-token\") pod \"e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:38.263112 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.263056 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mggx\" (UniqueName: \"kubernetes.io/projected/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-kube-api-access-5mggx\") pod \"e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:38.329260 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.329216 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:38.518599 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.518567 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s"] Jul 06 21:33:38.519660 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:33:38.519636 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee1fa7d3_4e66_49d1_aecb_65eed1559c64.slice/crio-d051969346c72884ea969c8c37e110d0b0b3482a3921cd666c7a666f81ae4610 WatchSource:0}: Error finding container d051969346c72884ea969c8c37e110d0b0b3482a3921cd666c7a666f81ae4610: Status 404 returned error can't find the container with id d051969346c72884ea969c8c37e110d0b0b3482a3921cd666c7a666f81ae4610 Jul 06 21:33:38.522465 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.522430 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:33:38.522577 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.522493 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:33:38.522577 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.522522 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:33:38.955657 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.955610 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" event={"ID":"ee1fa7d3-4e66-49d1-aecb-65eed1559c64","Type":"ContainerStarted","Data":"70f745b90fcd5fec9d51805e53a64e570302d07d582fecdddb6676100cf90ac5"} Jul 06 21:33:38.955856 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.955677 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" event={"ID":"ee1fa7d3-4e66-49d1-aecb-65eed1559c64","Type":"ContainerStarted","Data":"d051969346c72884ea969c8c37e110d0b0b3482a3921cd666c7a666f81ae4610"} Jul 06 21:33:38.979055 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:38.978972 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" podStartSLOduration=1.9789470009999999 podStartE2EDuration="1.978947001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:33:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:33:38.97713796 +0000 UTC m=+2103.295361244" watchObservedRunningTime="2026-07-06 21:33:38.978947001 +0000 UTC m=+2103.297170263" Jul 06 21:33:39.331637 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:39.331531 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:39.342383 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:39.342336 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:39.960618 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:39.960589 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:39.961624 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:39.961605 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:41.015833 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:41.015787 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4"] Jul 06 21:33:41.016608 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:41.016584 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d734cf1-9c31-4bf3-bed8-2c473389fa4b" containerName="istio-proxy" Jul 06 21:33:41.019849 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:41.019828 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4" Jul 06 21:33:41.022740 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:41.022716 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-5ea078-serving-cert\"" Jul 06 21:33:41.028268 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:41.028239 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4"] Jul 06 21:33:41.196293 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:41.196252 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b5640f7-fd5d-440e-bd07-5697315cda07-maas-api-tls\") pod \"maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4\" (UID: \"5b5640f7-fd5d-440e-bd07-5697315cda07\") " pod="opendatahub/maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4" Jul 06 21:33:41.196507 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:41.196325 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5640f7-fd5d-440e-bd07-5697315cda07-openshift-service-ca\") pod \"maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4\" (UID: \"5b5640f7-fd5d-440e-bd07-5697315cda07\") " pod="opendatahub/maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4" Jul 06 21:33:41.196507 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:41.196343 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckq2w\" (UniqueName: \"kubernetes.io/projected/5b5640f7-fd5d-440e-bd07-5697315cda07-kube-api-access-ckq2w\") pod \"maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4\" (UID: \"5b5640f7-fd5d-440e-bd07-5697315cda07\") " pod="opendatahub/maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4" Jul 06 21:33:41.297352 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:41.297243 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b5640f7-fd5d-440e-bd07-5697315cda07-maas-api-tls\") pod \"maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4\" (UID: \"5b5640f7-fd5d-440e-bd07-5697315cda07\") " pod="opendatahub/maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4" Jul 06 21:33:41.297352 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:41.297333 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5640f7-fd5d-440e-bd07-5697315cda07-openshift-service-ca\") pod \"maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4\" (UID: \"5b5640f7-fd5d-440e-bd07-5697315cda07\") " pod="opendatahub/maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4" Jul 06 21:33:41.297602 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:41.297363 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckq2w\" (UniqueName: \"kubernetes.io/projected/5b5640f7-fd5d-440e-bd07-5697315cda07-kube-api-access-ckq2w\") pod \"maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4\" (UID: \"5b5640f7-fd5d-440e-bd07-5697315cda07\") " pod="opendatahub/maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4" Jul 06 21:33:41.297602 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:33:41.297423 2577 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-b-5ea078-serving-cert: secret "maas-api-e2e-shared-b-5ea078-serving-cert" not found Jul 06 21:33:41.297602 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:33:41.297508 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b5640f7-fd5d-440e-bd07-5697315cda07-maas-api-tls podName:5b5640f7-fd5d-440e-bd07-5697315cda07 nodeName:}" failed. No retries permitted until 2026-07-06 21:33:41.797485626 +0000 UTC m=+2106.115708868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5b5640f7-fd5d-440e-bd07-5697315cda07-maas-api-tls") pod "maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4" (UID: "5b5640f7-fd5d-440e-bd07-5697315cda07") : secret "maas-api-e2e-shared-b-5ea078-serving-cert" not found Jul 06 21:33:41.298198 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:41.298173 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5640f7-fd5d-440e-bd07-5697315cda07-openshift-service-ca\") pod \"maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4\" (UID: \"5b5640f7-fd5d-440e-bd07-5697315cda07\") " pod="opendatahub/maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4" Jul 06 21:33:41.310692 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:41.310653 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckq2w\" (UniqueName: \"kubernetes.io/projected/5b5640f7-fd5d-440e-bd07-5697315cda07-kube-api-access-ckq2w\") pod \"maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4\" (UID: \"5b5640f7-fd5d-440e-bd07-5697315cda07\") " pod="opendatahub/maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4" Jul 06 21:33:41.803662 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:41.803625 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b5640f7-fd5d-440e-bd07-5697315cda07-maas-api-tls\") pod \"maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4\" (UID: \"5b5640f7-fd5d-440e-bd07-5697315cda07\") " pod="opendatahub/maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4" Jul 06 21:33:41.806912 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:41.806879 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b5640f7-fd5d-440e-bd07-5697315cda07-maas-api-tls\") pod \"maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4\" (UID: \"5b5640f7-fd5d-440e-bd07-5697315cda07\") " pod="opendatahub/maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4" Jul 06 21:33:41.960571 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:41.960535 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4" Jul 06 21:33:42.119238 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:42.119201 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4"] Jul 06 21:33:42.120392 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:33:42.120363 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b5640f7_fd5d_440e_bd07_5697315cda07.slice/crio-8e2f1a8c53479d42e92f39c44a2018e04a36b88ea2176a807e4a7d2b76b9f4ab WatchSource:0}: Error finding container 8e2f1a8c53479d42e92f39c44a2018e04a36b88ea2176a807e4a7d2b76b9f4ab: Status 404 returned error can't find the container with id 8e2f1a8c53479d42e92f39c44a2018e04a36b88ea2176a807e4a7d2b76b9f4ab Jul 06 21:33:42.980946 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:42.980910 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4" event={"ID":"5b5640f7-fd5d-440e-bd07-5697315cda07","Type":"ContainerStarted","Data":"00e7450804e8d6cd56e729954329408a2519f3f0cdf6d11af72b33d0124ef9e0"} Jul 06 21:33:42.981187 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:42.980967 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4" event={"ID":"5b5640f7-fd5d-440e-bd07-5697315cda07","Type":"ContainerStarted","Data":"8e2f1a8c53479d42e92f39c44a2018e04a36b88ea2176a807e4a7d2b76b9f4ab"} Jul 06 21:33:42.981187 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:42.980997 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4" Jul 06 21:33:43.001576 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:43.001517 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4" podStartSLOduration=2.680106787 podStartE2EDuration="3.001495966s" totalImagesPullingTime="321.389179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 21:33:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:33:42.999617852 +0000 UTC m=+2107.317841138" watchObservedRunningTime="2026-07-06 21:33:43.001495966 +0000 UTC m=+2107.319719237" Jul 06 21:33:44.993315 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:44.993234 2577 generic.go:356] "Generic (PLEG): container finished" podID="490125d5-40f7-468d-85f5-41d91b91bfc2" containerID="c422ce290a423ca5088de9b5a12b9cff5786080b957fb9934bcfcc9fc169e24c" exitCode=0 Jul 06 21:33:44.993315 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:44.993278 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" event={"ID":"490125d5-40f7-468d-85f5-41d91b91bfc2","Type":"ContainerDied","Data":"c422ce290a423ca5088de9b5a12b9cff5786080b957fb9934bcfcc9fc169e24c"} Jul 06 21:33:44.993747 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:44.993689 2577 scope.go:117] "RemoveContainer" containerID="c422ce290a423ca5088de9b5a12b9cff5786080b957fb9934bcfcc9fc169e24c" Jul 06 21:33:44.994726 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:44.994697 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:33:44.994821 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:44.994779 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:33:44.994883 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:44.994826 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:33:46.000138 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:46.000104 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" event={"ID":"490125d5-40f7-468d-85f5-41d91b91bfc2","Type":"ContainerStarted","Data":"33949161c1cacaed50c5af78d28175f935487800b42749df8cda3b01520c235b"} Jul 06 21:33:46.745368 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:46.745329 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:46.746864 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:46.746804 2577 patch_prober.go:28] interesting pod/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 06 21:33:46.747028 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:46.746912 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" podUID="490125d5-40f7-468d-85f5-41d91b91bfc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 06 21:33:46.909567 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:46.909532 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:47.744872 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:47.744823 2577 patch_prober.go:28] interesting pod/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 06 21:33:47.745401 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:47.744914 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" podUID="490125d5-40f7-468d-85f5-41d91b91bfc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 06 21:33:48.745104 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:48.745060 2577 patch_prober.go:28] interesting pod/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 06 21:33:48.745623 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:48.745129 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" podUID="490125d5-40f7-468d-85f5-41d91b91bfc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 06 21:33:48.992863 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:48.992827 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4" Jul 06 21:33:49.744505 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:49.744460 2577 patch_prober.go:28] interesting pod/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 06 21:33:49.744719 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:49.744543 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" podUID="490125d5-40f7-468d-85f5-41d91b91bfc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 06 21:33:50.753867 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:50.753830 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:50.755072 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:50.755050 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:33:56.002231 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:56.002184 2577 patch_prober.go:28] interesting pod/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 06 21:33:56.002651 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:56.002271 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" podUID="ee1fa7d3-4e66-49d1-aecb-65eed1559c64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 21:33:58.075023 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:58.074979 2577 generic.go:356] "Generic (PLEG): container finished" podID="ee1fa7d3-4e66-49d1-aecb-65eed1559c64" containerID="70f745b90fcd5fec9d51805e53a64e570302d07d582fecdddb6676100cf90ac5" exitCode=0 Jul 06 21:33:58.075448 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:58.075050 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" event={"ID":"ee1fa7d3-4e66-49d1-aecb-65eed1559c64","Type":"ContainerDied","Data":"70f745b90fcd5fec9d51805e53a64e570302d07d582fecdddb6676100cf90ac5"} Jul 06 21:33:58.075448 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:58.075423 2577 scope.go:117] "RemoveContainer" containerID="70f745b90fcd5fec9d51805e53a64e570302d07d582fecdddb6676100cf90ac5" Jul 06 21:33:58.076419 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:58.076391 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:33:58.076525 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:58.076461 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:33:58.076525 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:58.076491 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 21:33:59.081955 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:59.081920 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" event={"ID":"ee1fa7d3-4e66-49d1-aecb-65eed1559c64","Type":"ContainerStarted","Data":"f12e26d0e3075ba66d87bd164bd113161608ee3eac362160a24e4a7b39b1aff0"} Jul 06 21:33:59.329809 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:59.329763 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:33:59.330102 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:59.330070 2577 patch_prober.go:28] interesting pod/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 06 21:33:59.330198 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:33:59.330131 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" podUID="ee1fa7d3-4e66-49d1-aecb-65eed1559c64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 06 21:34:00.330131 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:34:00.330092 2577 patch_prober.go:28] interesting pod/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 06 21:34:00.330752 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:34:00.330717 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" podUID="ee1fa7d3-4e66-49d1-aecb-65eed1559c64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 06 21:34:01.330326 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:34:01.330282 2577 patch_prober.go:28] interesting pod/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 06 21:34:01.330945 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:34:01.330365 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" podUID="ee1fa7d3-4e66-49d1-aecb-65eed1559c64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 06 21:34:02.330249 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:34:02.330206 2577 patch_prober.go:28] interesting pod/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 06 21:34:02.330588 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:34:02.330551 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" podUID="ee1fa7d3-4e66-49d1-aecb-65eed1559c64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 06 21:34:03.333631 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:34:03.333599 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:34:03.334093 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:34:03.333886 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:34:03.334595 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:34:03.334580 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:38:36.365496 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:38:36.365464 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qjs2n_64b45669-ed56-41b9-a3ed-1fe5c13508b6/ovn-acl-logging/0.log" Jul 06 21:38:36.371428 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:38:36.371402 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qjs2n_64b45669-ed56-41b9-a3ed-1fe5c13508b6/ovn-acl-logging/0.log" Jul 06 21:39:45.251311 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:45.251268 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b"] Jul 06 21:39:45.251873 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:45.251579 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" podUID="490125d5-40f7-468d-85f5-41d91b91bfc2" containerName="istio-proxy" containerID="cri-o://33949161c1cacaed50c5af78d28175f935487800b42749df8cda3b01520c235b" gracePeriod=30 Jul 06 21:39:45.993850 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:45.993813 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4"] Jul 06 21:39:45.994089 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:45.994048 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4" podUID="5b5640f7-fd5d-440e-bd07-5697315cda07" containerName="maas-api" containerID="cri-o://00e7450804e8d6cd56e729954329408a2519f3f0cdf6d11af72b33d0124ef9e0" gracePeriod=30 Jul 06 21:39:46.259479 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:46.259448 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4" Jul 06 21:39:46.270237 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:46.270205 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b5640f7-fd5d-440e-bd07-5697315cda07-maas-api-tls\") pod \"5b5640f7-fd5d-440e-bd07-5697315cda07\" (UID: \"5b5640f7-fd5d-440e-bd07-5697315cda07\") " Jul 06 21:39:46.270424 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:46.270327 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5640f7-fd5d-440e-bd07-5697315cda07-openshift-service-ca\") pod \"5b5640f7-fd5d-440e-bd07-5697315cda07\" (UID: \"5b5640f7-fd5d-440e-bd07-5697315cda07\") " Jul 06 21:39:46.270424 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:46.270353 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckq2w\" (UniqueName: \"kubernetes.io/projected/5b5640f7-fd5d-440e-bd07-5697315cda07-kube-api-access-ckq2w\") pod \"5b5640f7-fd5d-440e-bd07-5697315cda07\" (UID: \"5b5640f7-fd5d-440e-bd07-5697315cda07\") " Jul 06 21:39:46.270716 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:46.270695 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b5640f7-fd5d-440e-bd07-5697315cda07-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5b5640f7-fd5d-440e-bd07-5697315cda07" (UID: "5b5640f7-fd5d-440e-bd07-5697315cda07"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:39:46.272622 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:46.272589 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b5640f7-fd5d-440e-bd07-5697315cda07-kube-api-access-ckq2w" (OuterVolumeSpecName: "kube-api-access-ckq2w") pod "5b5640f7-fd5d-440e-bd07-5697315cda07" (UID: "5b5640f7-fd5d-440e-bd07-5697315cda07"). InnerVolumeSpecName "kube-api-access-ckq2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:39:46.272622 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:46.272590 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b5640f7-fd5d-440e-bd07-5697315cda07-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5b5640f7-fd5d-440e-bd07-5697315cda07" (UID: "5b5640f7-fd5d-440e-bd07-5697315cda07"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 21:39:46.371457 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:46.371419 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b5640f7-fd5d-440e-bd07-5697315cda07-maas-api-tls\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:39:46.371457 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:46.371450 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5640f7-fd5d-440e-bd07-5697315cda07-openshift-service-ca\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:39:46.371457 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:46.371459 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckq2w\" (UniqueName: \"kubernetes.io/projected/5b5640f7-fd5d-440e-bd07-5697315cda07-kube-api-access-ckq2w\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:39:46.987194 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:46.987155 2577 generic.go:356] "Generic (PLEG): container finished" podID="5b5640f7-fd5d-440e-bd07-5697315cda07" containerID="00e7450804e8d6cd56e729954329408a2519f3f0cdf6d11af72b33d0124ef9e0" exitCode=0 Jul 06 21:39:46.987194 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:46.987198 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4" event={"ID":"5b5640f7-fd5d-440e-bd07-5697315cda07","Type":"ContainerDied","Data":"00e7450804e8d6cd56e729954329408a2519f3f0cdf6d11af72b33d0124ef9e0"} Jul 06 21:39:46.987474 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:46.987200 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4" Jul 06 21:39:46.987474 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:46.987249 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4" event={"ID":"5b5640f7-fd5d-440e-bd07-5697315cda07","Type":"ContainerDied","Data":"8e2f1a8c53479d42e92f39c44a2018e04a36b88ea2176a807e4a7d2b76b9f4ab"} Jul 06 21:39:46.987474 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:46.987275 2577 scope.go:117] "RemoveContainer" containerID="00e7450804e8d6cd56e729954329408a2519f3f0cdf6d11af72b33d0124ef9e0" Jul 06 21:39:46.997701 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:46.997681 2577 scope.go:117] "RemoveContainer" containerID="00e7450804e8d6cd56e729954329408a2519f3f0cdf6d11af72b33d0124ef9e0" Jul 06 21:39:46.998050 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:39:46.998023 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00e7450804e8d6cd56e729954329408a2519f3f0cdf6d11af72b33d0124ef9e0\": container with ID starting with 00e7450804e8d6cd56e729954329408a2519f3f0cdf6d11af72b33d0124ef9e0 not found: ID does not exist" containerID="00e7450804e8d6cd56e729954329408a2519f3f0cdf6d11af72b33d0124ef9e0" Jul 06 21:39:46.998158 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:46.998061 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e7450804e8d6cd56e729954329408a2519f3f0cdf6d11af72b33d0124ef9e0"} err="failed to get container status \"00e7450804e8d6cd56e729954329408a2519f3f0cdf6d11af72b33d0124ef9e0\": rpc error: code = NotFound desc = could not find container \"00e7450804e8d6cd56e729954329408a2519f3f0cdf6d11af72b33d0124ef9e0\": container with ID starting with 00e7450804e8d6cd56e729954329408a2519f3f0cdf6d11af72b33d0124ef9e0 not found: ID does not exist" Jul 06 21:39:47.014936 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:47.014891 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4"] Jul 06 21:39:47.017439 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:47.017408 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-5ea078-7b8dccd96-mbvn4"] Jul 06 21:39:48.168735 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:48.168694 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b5640f7-fd5d-440e-bd07-5697315cda07" path="/var/lib/kubelet/pods/5b5640f7-fd5d-440e-bd07-5697315cda07/volumes" Jul 06 21:39:50.516909 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:50.516838 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:39:50.611904 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:50.611865 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwwpl\" (UniqueName: \"kubernetes.io/projected/490125d5-40f7-468d-85f5-41d91b91bfc2-kube-api-access-mwwpl\") pod \"490125d5-40f7-468d-85f5-41d91b91bfc2\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " Jul 06 21:39:50.611904 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:50.611913 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/490125d5-40f7-468d-85f5-41d91b91bfc2-credential-socket\") pod \"490125d5-40f7-468d-85f5-41d91b91bfc2\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " Jul 06 21:39:50.612222 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:50.611931 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/490125d5-40f7-468d-85f5-41d91b91bfc2-istio-podinfo\") pod \"490125d5-40f7-468d-85f5-41d91b91bfc2\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " Jul 06 21:39:50.612222 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:50.611957 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/490125d5-40f7-468d-85f5-41d91b91bfc2-istio-data\") pod \"490125d5-40f7-468d-85f5-41d91b91bfc2\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " Jul 06 21:39:50.612222 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:50.611999 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/490125d5-40f7-468d-85f5-41d91b91bfc2-istio-envoy\") pod \"490125d5-40f7-468d-85f5-41d91b91bfc2\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " Jul 06 21:39:50.612222 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:50.612077 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/490125d5-40f7-468d-85f5-41d91b91bfc2-istio-token\") pod \"490125d5-40f7-468d-85f5-41d91b91bfc2\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " Jul 06 21:39:50.612222 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:50.612114 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/490125d5-40f7-468d-85f5-41d91b91bfc2-workload-socket\") pod \"490125d5-40f7-468d-85f5-41d91b91bfc2\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " Jul 06 21:39:50.612222 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:50.612153 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/490125d5-40f7-468d-85f5-41d91b91bfc2-workload-certs\") pod \"490125d5-40f7-468d-85f5-41d91b91bfc2\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " Jul 06 21:39:50.612222 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:50.612174 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/490125d5-40f7-468d-85f5-41d91b91bfc2-istiod-ca-cert\") pod \"490125d5-40f7-468d-85f5-41d91b91bfc2\" (UID: \"490125d5-40f7-468d-85f5-41d91b91bfc2\") " Jul 06 21:39:50.612585 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:50.612397 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/490125d5-40f7-468d-85f5-41d91b91bfc2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "490125d5-40f7-468d-85f5-41d91b91bfc2" (UID: "490125d5-40f7-468d-85f5-41d91b91bfc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:39:50.612585 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:50.612539 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/490125d5-40f7-468d-85f5-41d91b91bfc2-credential-socket\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:39:50.612585 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:50.612521 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/490125d5-40f7-468d-85f5-41d91b91bfc2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "490125d5-40f7-468d-85f5-41d91b91bfc2" (UID: "490125d5-40f7-468d-85f5-41d91b91bfc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:39:50.612744 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:50.612698 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/490125d5-40f7-468d-85f5-41d91b91bfc2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "490125d5-40f7-468d-85f5-41d91b91bfc2" (UID: "490125d5-40f7-468d-85f5-41d91b91bfc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:39:50.612744 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:50.612715 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/490125d5-40f7-468d-85f5-41d91b91bfc2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "490125d5-40f7-468d-85f5-41d91b91bfc2" (UID: "490125d5-40f7-468d-85f5-41d91b91bfc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:39:50.613047 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:50.612988 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/490125d5-40f7-468d-85f5-41d91b91bfc2-istio-data" (OuterVolumeSpecName: "istio-data") pod "490125d5-40f7-468d-85f5-41d91b91bfc2" (UID: "490125d5-40f7-468d-85f5-41d91b91bfc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:39:50.614418 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:50.614387 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/490125d5-40f7-468d-85f5-41d91b91bfc2-istio-token" (OuterVolumeSpecName: "istio-token") pod "490125d5-40f7-468d-85f5-41d91b91bfc2" (UID: "490125d5-40f7-468d-85f5-41d91b91bfc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:39:50.614550 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:50.614421 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/490125d5-40f7-468d-85f5-41d91b91bfc2-kube-api-access-mwwpl" (OuterVolumeSpecName: "kube-api-access-mwwpl") pod "490125d5-40f7-468d-85f5-41d91b91bfc2" (UID: "490125d5-40f7-468d-85f5-41d91b91bfc2"). InnerVolumeSpecName "kube-api-access-mwwpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:39:50.614722 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:50.614697 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/490125d5-40f7-468d-85f5-41d91b91bfc2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "490125d5-40f7-468d-85f5-41d91b91bfc2" (UID: "490125d5-40f7-468d-85f5-41d91b91bfc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 21:39:50.614814 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:50.614760 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/490125d5-40f7-468d-85f5-41d91b91bfc2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "490125d5-40f7-468d-85f5-41d91b91bfc2" (UID: "490125d5-40f7-468d-85f5-41d91b91bfc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:39:50.713051 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:50.712987 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/490125d5-40f7-468d-85f5-41d91b91bfc2-istio-envoy\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:39:50.713051 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:50.713043 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/490125d5-40f7-468d-85f5-41d91b91bfc2-istio-token\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:39:50.713051 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:50.713053 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/490125d5-40f7-468d-85f5-41d91b91bfc2-workload-socket\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:39:50.713051 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:50.713063 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/490125d5-40f7-468d-85f5-41d91b91bfc2-workload-certs\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:39:50.713321 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:50.713072 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/490125d5-40f7-468d-85f5-41d91b91bfc2-istiod-ca-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:39:50.713321 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:50.713080 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwwpl\" (UniqueName: \"kubernetes.io/projected/490125d5-40f7-468d-85f5-41d91b91bfc2-kube-api-access-mwwpl\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:39:50.713321 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:50.713089 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/490125d5-40f7-468d-85f5-41d91b91bfc2-istio-podinfo\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:39:50.713321 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:50.713097 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/490125d5-40f7-468d-85f5-41d91b91bfc2-istio-data\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:39:51.010380 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:51.010346 2577 generic.go:356] "Generic (PLEG): container finished" podID="490125d5-40f7-468d-85f5-41d91b91bfc2" containerID="33949161c1cacaed50c5af78d28175f935487800b42749df8cda3b01520c235b" exitCode=0 Jul 06 21:39:51.010529 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:51.010418 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" Jul 06 21:39:51.010529 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:51.010425 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" event={"ID":"490125d5-40f7-468d-85f5-41d91b91bfc2","Type":"ContainerDied","Data":"33949161c1cacaed50c5af78d28175f935487800b42749df8cda3b01520c235b"} Jul 06 21:39:51.010529 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:51.010503 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b" event={"ID":"490125d5-40f7-468d-85f5-41d91b91bfc2","Type":"ContainerDied","Data":"c51236bc1571e9a57f48ac9051deab671fe9853235da7af6bd68ae55ea6c83ae"} Jul 06 21:39:51.010529 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:51.010526 2577 scope.go:117] "RemoveContainer" containerID="33949161c1cacaed50c5af78d28175f935487800b42749df8cda3b01520c235b" Jul 06 21:39:51.021201 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:51.021181 2577 scope.go:117] "RemoveContainer" containerID="c422ce290a423ca5088de9b5a12b9cff5786080b957fb9934bcfcc9fc169e24c" Jul 06 21:39:51.029776 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:51.029754 2577 scope.go:117] "RemoveContainer" containerID="33949161c1cacaed50c5af78d28175f935487800b42749df8cda3b01520c235b" Jul 06 21:39:51.030111 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:39:51.030089 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33949161c1cacaed50c5af78d28175f935487800b42749df8cda3b01520c235b\": container with ID starting with 33949161c1cacaed50c5af78d28175f935487800b42749df8cda3b01520c235b not found: ID does not exist" containerID="33949161c1cacaed50c5af78d28175f935487800b42749df8cda3b01520c235b" Jul 06 21:39:51.030215 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:51.030125 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33949161c1cacaed50c5af78d28175f935487800b42749df8cda3b01520c235b"} err="failed to get container status \"33949161c1cacaed50c5af78d28175f935487800b42749df8cda3b01520c235b\": rpc error: code = NotFound desc = could not find container \"33949161c1cacaed50c5af78d28175f935487800b42749df8cda3b01520c235b\": container with ID starting with 33949161c1cacaed50c5af78d28175f935487800b42749df8cda3b01520c235b not found: ID does not exist" Jul 06 21:39:51.030215 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:51.030151 2577 scope.go:117] "RemoveContainer" containerID="c422ce290a423ca5088de9b5a12b9cff5786080b957fb9934bcfcc9fc169e24c" Jul 06 21:39:51.030502 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:39:51.030474 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c422ce290a423ca5088de9b5a12b9cff5786080b957fb9934bcfcc9fc169e24c\": container with ID starting with c422ce290a423ca5088de9b5a12b9cff5786080b957fb9934bcfcc9fc169e24c not found: ID does not exist" containerID="c422ce290a423ca5088de9b5a12b9cff5786080b957fb9934bcfcc9fc169e24c" Jul 06 21:39:51.030603 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:51.030510 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c422ce290a423ca5088de9b5a12b9cff5786080b957fb9934bcfcc9fc169e24c"} err="failed to get container status \"c422ce290a423ca5088de9b5a12b9cff5786080b957fb9934bcfcc9fc169e24c\": rpc error: code = NotFound desc = could not find container \"c422ce290a423ca5088de9b5a12b9cff5786080b957fb9934bcfcc9fc169e24c\": container with ID starting with c422ce290a423ca5088de9b5a12b9cff5786080b957fb9934bcfcc9fc169e24c not found: ID does not exist" Jul 06 21:39:51.033339 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:51.033315 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b"] Jul 06 21:39:51.039326 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:51.039299 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-900ebe-openshift-default-d7cdf6c75-mt98b"] Jul 06 21:39:52.169561 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:52.169528 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="490125d5-40f7-468d-85f5-41d91b91bfc2" path="/var/lib/kubelet/pods/490125d5-40f7-468d-85f5-41d91b91bfc2/volumes" Jul 06 21:39:52.458807 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:52.458710 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s"] Jul 06 21:39:52.459034 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:52.458988 2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" podUID="ee1fa7d3-4e66-49d1-aecb-65eed1559c64" containerName="istio-proxy" containerID="cri-o://f12e26d0e3075ba66d87bd164bd113161608ee3eac362160a24e4a7b39b1aff0" gracePeriod=30 Jul 06 21:39:57.725980 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:57.725913 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:39:57.789474 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:57.789440 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-credential-socket\") pod \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " Jul 06 21:39:57.789670 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:57.789513 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mggx\" (UniqueName: \"kubernetes.io/projected/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-kube-api-access-5mggx\") pod \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " Jul 06 21:39:57.789670 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:57.789551 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-istio-data\") pod \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " Jul 06 21:39:57.789670 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:57.789566 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-istio-token\") pod \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " Jul 06 21:39:57.789670 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:57.789599 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-workload-certs\") pod \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " Jul 06 21:39:57.789670 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:57.789664 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-istiod-ca-cert\") pod \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " Jul 06 21:39:57.789945 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:57.789716 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-istio-envoy\") pod \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " Jul 06 21:39:57.789945 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:57.789740 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-workload-socket\") pod \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " Jul 06 21:39:57.789945 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:57.789793 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-istio-podinfo\") pod \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\" (UID: \"ee1fa7d3-4e66-49d1-aecb-65eed1559c64\") " Jul 06 21:39:57.789945 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:57.789837 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ee1fa7d3-4e66-49d1-aecb-65eed1559c64" (UID: "ee1fa7d3-4e66-49d1-aecb-65eed1559c64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:39:57.789945 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:57.789921 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ee1fa7d3-4e66-49d1-aecb-65eed1559c64" (UID: "ee1fa7d3-4e66-49d1-aecb-65eed1559c64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:39:57.790241 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:57.790140 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ee1fa7d3-4e66-49d1-aecb-65eed1559c64" (UID: "ee1fa7d3-4e66-49d1-aecb-65eed1559c64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 21:39:57.791845 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:57.790236 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-credential-socket\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:39:57.791845 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:57.790256 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-workload-certs\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:39:57.791845 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:57.790270 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-istiod-ca-cert\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:39:57.791845 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:57.790562 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ee1fa7d3-4e66-49d1-aecb-65eed1559c64" (UID: "ee1fa7d3-4e66-49d1-aecb-65eed1559c64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:39:57.791845 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:57.791498 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-istio-data" (OuterVolumeSpecName: "istio-data") pod "ee1fa7d3-4e66-49d1-aecb-65eed1559c64" (UID: "ee1fa7d3-4e66-49d1-aecb-65eed1559c64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:39:57.792454 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:57.792419 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-istio-token" (OuterVolumeSpecName: "istio-token") pod "ee1fa7d3-4e66-49d1-aecb-65eed1559c64" (UID: "ee1fa7d3-4e66-49d1-aecb-65eed1559c64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:39:57.792862 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:57.792838 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-kube-api-access-5mggx" (OuterVolumeSpecName: "kube-api-access-5mggx") pod "ee1fa7d3-4e66-49d1-aecb-65eed1559c64" (UID: "ee1fa7d3-4e66-49d1-aecb-65eed1559c64"). InnerVolumeSpecName "kube-api-access-5mggx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 21:39:57.793049 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:57.793004 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ee1fa7d3-4e66-49d1-aecb-65eed1559c64" (UID: "ee1fa7d3-4e66-49d1-aecb-65eed1559c64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 21:39:57.793213 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:57.793195 2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ee1fa7d3-4e66-49d1-aecb-65eed1559c64" (UID: "ee1fa7d3-4e66-49d1-aecb-65eed1559c64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 21:39:57.890811 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:57.890770 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-istio-envoy\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:39:57.890811 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:57.890805 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-workload-socket\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:39:57.890811 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:57.890815 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-istio-podinfo\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:39:57.891095 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:57.890823 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mggx\" (UniqueName: \"kubernetes.io/projected/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-kube-api-access-5mggx\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:39:57.891095 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:57.890835 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-istio-data\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:39:57.891095 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:57.890842 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1fa7d3-4e66-49d1-aecb-65eed1559c64-istio-token\") on node \"ip-10-0-136-207.ec2.internal\" DevicePath \"\"" Jul 06 21:39:58.050303 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:58.050209 2577 generic.go:356] "Generic (PLEG): container finished" podID="ee1fa7d3-4e66-49d1-aecb-65eed1559c64" containerID="f12e26d0e3075ba66d87bd164bd113161608ee3eac362160a24e4a7b39b1aff0" exitCode=0 Jul 06 21:39:58.050303 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:58.050262 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" Jul 06 21:39:58.050303 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:58.050277 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" event={"ID":"ee1fa7d3-4e66-49d1-aecb-65eed1559c64","Type":"ContainerDied","Data":"f12e26d0e3075ba66d87bd164bd113161608ee3eac362160a24e4a7b39b1aff0"} Jul 06 21:39:58.050528 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:58.050328 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s" event={"ID":"ee1fa7d3-4e66-49d1-aecb-65eed1559c64","Type":"ContainerDied","Data":"d051969346c72884ea969c8c37e110d0b0b3482a3921cd666c7a666f81ae4610"} Jul 06 21:39:58.050528 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:58.050350 2577 scope.go:117] "RemoveContainer" containerID="f12e26d0e3075ba66d87bd164bd113161608ee3eac362160a24e4a7b39b1aff0" Jul 06 21:39:58.059773 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:58.059752 2577 scope.go:117] "RemoveContainer" containerID="70f745b90fcd5fec9d51805e53a64e570302d07d582fecdddb6676100cf90ac5" Jul 06 21:39:58.068124 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:58.068055 2577 scope.go:117] "RemoveContainer" containerID="f12e26d0e3075ba66d87bd164bd113161608ee3eac362160a24e4a7b39b1aff0" Jul 06 21:39:58.068415 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:39:58.068392 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f12e26d0e3075ba66d87bd164bd113161608ee3eac362160a24e4a7b39b1aff0\": container with ID starting with f12e26d0e3075ba66d87bd164bd113161608ee3eac362160a24e4a7b39b1aff0 not found: ID does not exist" containerID="f12e26d0e3075ba66d87bd164bd113161608ee3eac362160a24e4a7b39b1aff0" Jul 06 21:39:58.068521 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:58.068423 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f12e26d0e3075ba66d87bd164bd113161608ee3eac362160a24e4a7b39b1aff0"} err="failed to get container status \"f12e26d0e3075ba66d87bd164bd113161608ee3eac362160a24e4a7b39b1aff0\": rpc error: code = NotFound desc = could not find container \"f12e26d0e3075ba66d87bd164bd113161608ee3eac362160a24e4a7b39b1aff0\": container with ID starting with f12e26d0e3075ba66d87bd164bd113161608ee3eac362160a24e4a7b39b1aff0 not found: ID does not exist" Jul 06 21:39:58.068521 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:58.068449 2577 scope.go:117] "RemoveContainer" containerID="70f745b90fcd5fec9d51805e53a64e570302d07d582fecdddb6676100cf90ac5" Jul 06 21:39:58.068769 ip-10-0-136-207 kubenswrapper[2577]: E0706 21:39:58.068735 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70f745b90fcd5fec9d51805e53a64e570302d07d582fecdddb6676100cf90ac5\": container with ID starting with 70f745b90fcd5fec9d51805e53a64e570302d07d582fecdddb6676100cf90ac5 not found: ID does not exist" containerID="70f745b90fcd5fec9d51805e53a64e570302d07d582fecdddb6676100cf90ac5" Jul 06 21:39:58.068839 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:58.068762 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70f745b90fcd5fec9d51805e53a64e570302d07d582fecdddb6676100cf90ac5"} err="failed to get container status \"70f745b90fcd5fec9d51805e53a64e570302d07d582fecdddb6676100cf90ac5\": rpc error: code = NotFound desc = could not find container \"70f745b90fcd5fec9d51805e53a64e570302d07d582fecdddb6676100cf90ac5\": container with ID starting with 70f745b90fcd5fec9d51805e53a64e570302d07d582fecdddb6676100cf90ac5 not found: ID does not exist" Jul 06 21:39:58.076314 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:58.076290 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s"] Jul 06 21:39:58.078135 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:58.078114 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-5ea078-openshift-default-66d4846d7-9mv7s"] Jul 06 21:39:58.169166 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:58.169132 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee1fa7d3-4e66-49d1-aecb-65eed1559c64" path="/var/lib/kubelet/pods/ee1fa7d3-4e66-49d1-aecb-65eed1559c64/volumes" Jul 06 21:39:58.226705 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:58.226674 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-688cc6cd9f-d78bw_013bc0d1-86de-4f5e-bce5-04c3e39798fa/maas-api/0.log" Jul 06 21:39:58.347590 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:58.347501 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722875-nmtng_ce7f9d1b-993b-44dc-9972-e708a95283d9/cleanup/0.log" Jul 06 21:39:58.467985 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:58.467958 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722890-lmd47_303aa271-951a-4bbd-aa49-33adbf5b4d28/cleanup/0.log" Jul 06 21:39:58.592718 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:58.592689 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67f746bd7b-tgt54_406c8c90-2de7-4516-b7e0-22a23bc828f2/manager/0.log" Jul 06 21:39:58.966426 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:58.966393 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bd9f97986-7ht5r_bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d/manager/0.log" Jul 06 21:39:58.973761 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:58.973732 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bd9f97986-7ht5r_bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d/copy-manifests/0.log" Jul 06 21:39:59.367696 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:39:59.367619 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-r9pjt_fdc5952f-0f86-4a51-8678-d1887ab25ac0/postgres/0.log" Jul 06 21:40:00.833611 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:00.833580 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-wtnzk_04a627be-8e89-407d-9065-5ede85ce9e8a/manager/0.log" Jul 06 21:40:01.204844 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:01.204796 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-rtw2l_b5119452-a047-4283-9b62-b1f37fe2032c/registry-server/0.log" Jul 06 21:40:01.928991 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:01.928957 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4jr2lb_980321c5-434c-4e39-ae74-2c4c5b1537b8/istio-proxy/0.log" Jul 06 21:40:02.396816 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:02.396780 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-77zzd_90be9695-5aa7-4767-95a3-81acb10d1237/istio-proxy/1.log" Jul 06 21:40:02.521641 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:02.521601 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-nrqcv_e6c09920-f5fb-4db5-b5de-06d64bfb48f9/payload-pre-processing/0.log" Jul 06 21:40:02.648359 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:02.648273 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-sk8mt_c4dad928-112f-4baf-a79d-c9294ddadbd1/payload-processing/0.log" Jul 06 21:40:03.246753 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:03.246714 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-tzkfx_a684730a-371b-44c5-8798-e6db60d97413/main/0.log" Jul 06 21:40:03.375940 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:03.375898 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-8wbq9_976f5cf5-f24c-4578-aa50-beeafa06330f/main/0.log" Jul 06 21:40:11.113859 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:11.113826 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dbnt9_b0f9a2ff-6f72-4374-a459-b4da598d3533/global-pull-secret-syncer/0.log" Jul 06 21:40:11.359519 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:11.359486 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pvvfl_8ca96cb1-d98e-4362-b6ad-00dd336bd23d/konnectivity-agent/0.log" Jul 06 21:40:11.419185 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:11.419141 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-207.ec2.internal_835fe22bba349376f78a29c5926e82d5/haproxy/0.log" Jul 06 21:40:16.676108 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:16.676073 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-wtnzk_04a627be-8e89-407d-9065-5ede85ce9e8a/manager/0.log" Jul 06 21:40:16.790731 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:16.790694 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-rtw2l_b5119452-a047-4283-9b62-b1f37fe2032c/registry-server/0.log" Jul 06 21:40:18.728622 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:18.728530 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0f857be8-1333-4976-8193-892304f05ff3/alertmanager/0.log" Jul 06 21:40:18.763827 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:18.763796 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0f857be8-1333-4976-8193-892304f05ff3/config-reloader/0.log" Jul 06 21:40:18.795273 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:18.795243 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0f857be8-1333-4976-8193-892304f05ff3/kube-rbac-proxy-web/0.log" Jul 06 21:40:18.820631 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:18.820598 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0f857be8-1333-4976-8193-892304f05ff3/kube-rbac-proxy/0.log" Jul 06 21:40:18.846315 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:18.846271 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0f857be8-1333-4976-8193-892304f05ff3/kube-rbac-proxy-metric/0.log" Jul 06 21:40:18.873750 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:18.873722 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0f857be8-1333-4976-8193-892304f05ff3/prom-label-proxy/0.log" Jul 06 21:40:18.900028 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:18.899983 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0f857be8-1333-4976-8193-892304f05ff3/init-config-reloader/0.log" Jul 06 21:40:19.091192 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:19.091096 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-dhcnh_0ab3ede5-c596-4fab-a7db-683fa6c81ebc/kube-state-metrics/0.log" Jul 06 21:40:19.109058 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:19.109026 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-dhcnh_0ab3ede5-c596-4fab-a7db-683fa6c81ebc/kube-rbac-proxy-main/0.log" Jul 06 21:40:19.135146 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:19.135118 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-dhcnh_0ab3ede5-c596-4fab-a7db-683fa6c81ebc/kube-rbac-proxy-self/0.log" Jul 06 21:40:19.196739 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:19.196704 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-r2hp9_374e460c-d8aa-4a91-972b-2cf27ff28dc2/monitoring-plugin/0.log" Jul 06 21:40:19.324993 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:19.324965 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-785dd_536941ec-7aff-42ca-b8d9-fba6959b98aa/node-exporter/0.log" Jul 06 21:40:19.348002 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:19.347917 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-785dd_536941ec-7aff-42ca-b8d9-fba6959b98aa/kube-rbac-proxy/0.log" Jul 06 21:40:19.373055 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:19.373000 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-785dd_536941ec-7aff-42ca-b8d9-fba6959b98aa/init-textfile/0.log" Jul 06 21:40:19.590992 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:19.590958 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_80b3f415-31f0-400d-a51b-b905f18fd8a3/prometheus/0.log" Jul 06 21:40:19.615115 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:19.615028 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_80b3f415-31f0-400d-a51b-b905f18fd8a3/config-reloader/0.log" Jul 06 21:40:19.638599 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:19.638564 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_80b3f415-31f0-400d-a51b-b905f18fd8a3/thanos-sidecar/0.log" Jul 06 21:40:19.662769 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:19.662743 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_80b3f415-31f0-400d-a51b-b905f18fd8a3/kube-rbac-proxy-web/0.log" Jul 06 21:40:19.688769 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:19.688741 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_80b3f415-31f0-400d-a51b-b905f18fd8a3/kube-rbac-proxy/0.log" Jul 06 21:40:19.714173 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:19.714131 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_80b3f415-31f0-400d-a51b-b905f18fd8a3/kube-rbac-proxy-thanos/0.log" Jul 06 21:40:19.738836 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:19.738807 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_80b3f415-31f0-400d-a51b-b905f18fd8a3/init-config-reloader/0.log" Jul 06 21:40:21.272495 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.272467 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-m5kns_7c85aee9-0db9-43f1-8011-f04079c0bce8/networking-console-plugin/0.log" Jul 06 21:40:21.573342 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.573251 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-64qs2/perf-node-gather-daemonset-gfq65"] Jul 06 21:40:21.573647 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.573634 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="490125d5-40f7-468d-85f5-41d91b91bfc2" containerName="istio-proxy" Jul 06 21:40:21.573690 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.573648 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="490125d5-40f7-468d-85f5-41d91b91bfc2" containerName="istio-proxy" Jul 06 21:40:21.573690 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.573656 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee1fa7d3-4e66-49d1-aecb-65eed1559c64" containerName="istio-proxy" Jul 06 21:40:21.573690 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.573661 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee1fa7d3-4e66-49d1-aecb-65eed1559c64" containerName="istio-proxy" Jul 06 21:40:21.573690 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.573672 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b5640f7-fd5d-440e-bd07-5697315cda07" containerName="maas-api" Jul 06 21:40:21.573690 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.573677 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b5640f7-fd5d-440e-bd07-5697315cda07" containerName="maas-api" Jul 06 21:40:21.573690 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.573691 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="490125d5-40f7-468d-85f5-41d91b91bfc2" containerName="istio-proxy" Jul 06 21:40:21.573873 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.573696 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="490125d5-40f7-468d-85f5-41d91b91bfc2" containerName="istio-proxy" Jul 06 21:40:21.573873 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.573768 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee1fa7d3-4e66-49d1-aecb-65eed1559c64" containerName="istio-proxy" Jul 06 21:40:21.573873 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.573775 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="490125d5-40f7-468d-85f5-41d91b91bfc2" containerName="istio-proxy" Jul 06 21:40:21.573873 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.573782 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="490125d5-40f7-468d-85f5-41d91b91bfc2" containerName="istio-proxy" Jul 06 21:40:21.573873 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.573787 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b5640f7-fd5d-440e-bd07-5697315cda07" containerName="maas-api" Jul 06 21:40:21.573873 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.573793 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee1fa7d3-4e66-49d1-aecb-65eed1559c64" containerName="istio-proxy" Jul 06 21:40:21.576871 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.576844 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-64qs2/perf-node-gather-daemonset-gfq65" Jul 06 21:40:21.579561 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.579537 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-64qs2\"/\"openshift-service-ca.crt\"" Jul 06 21:40:21.579726 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.579710 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-64qs2\"/\"kube-root-ca.crt\"" Jul 06 21:40:21.580404 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.580390 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-64qs2\"/\"default-dockercfg-hh7hq\"" Jul 06 21:40:21.588098 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.588067 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-64qs2/perf-node-gather-daemonset-gfq65"] Jul 06 21:40:21.618996 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.618961 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcfv8\" (UniqueName: \"kubernetes.io/projected/b3c3fe09-707c-4590-a7bd-6203c4d4cd00-kube-api-access-lcfv8\") pod \"perf-node-gather-daemonset-gfq65\" (UID: \"b3c3fe09-707c-4590-a7bd-6203c4d4cd00\") " pod="openshift-must-gather-64qs2/perf-node-gather-daemonset-gfq65" Jul 06 21:40:21.618996 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.619025 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b3c3fe09-707c-4590-a7bd-6203c4d4cd00-sys\") pod \"perf-node-gather-daemonset-gfq65\" (UID: \"b3c3fe09-707c-4590-a7bd-6203c4d4cd00\") " pod="openshift-must-gather-64qs2/perf-node-gather-daemonset-gfq65" Jul 06 21:40:21.619247 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.619061 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b3c3fe09-707c-4590-a7bd-6203c4d4cd00-lib-modules\") pod \"perf-node-gather-daemonset-gfq65\" (UID: \"b3c3fe09-707c-4590-a7bd-6203c4d4cd00\") " pod="openshift-must-gather-64qs2/perf-node-gather-daemonset-gfq65" Jul 06 21:40:21.619247 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.619081 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b3c3fe09-707c-4590-a7bd-6203c4d4cd00-podres\") pod \"perf-node-gather-daemonset-gfq65\" (UID: \"b3c3fe09-707c-4590-a7bd-6203c4d4cd00\") " pod="openshift-must-gather-64qs2/perf-node-gather-daemonset-gfq65" Jul 06 21:40:21.619247 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.619133 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b3c3fe09-707c-4590-a7bd-6203c4d4cd00-proc\") pod \"perf-node-gather-daemonset-gfq65\" (UID: \"b3c3fe09-707c-4590-a7bd-6203c4d4cd00\") " pod="openshift-must-gather-64qs2/perf-node-gather-daemonset-gfq65" Jul 06 21:40:21.719915 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.719873 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b3c3fe09-707c-4590-a7bd-6203c4d4cd00-sys\") pod \"perf-node-gather-daemonset-gfq65\" (UID: \"b3c3fe09-707c-4590-a7bd-6203c4d4cd00\") " pod="openshift-must-gather-64qs2/perf-node-gather-daemonset-gfq65" Jul 06 21:40:21.720130 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.719929 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b3c3fe09-707c-4590-a7bd-6203c4d4cd00-lib-modules\") pod \"perf-node-gather-daemonset-gfq65\" (UID: \"b3c3fe09-707c-4590-a7bd-6203c4d4cd00\") " pod="openshift-must-gather-64qs2/perf-node-gather-daemonset-gfq65" Jul 06 21:40:21.720130 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.719959 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b3c3fe09-707c-4590-a7bd-6203c4d4cd00-podres\") pod \"perf-node-gather-daemonset-gfq65\" (UID: \"b3c3fe09-707c-4590-a7bd-6203c4d4cd00\") " pod="openshift-must-gather-64qs2/perf-node-gather-daemonset-gfq65" Jul 06 21:40:21.720130 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.719987 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b3c3fe09-707c-4590-a7bd-6203c4d4cd00-proc\") pod \"perf-node-gather-daemonset-gfq65\" (UID: \"b3c3fe09-707c-4590-a7bd-6203c4d4cd00\") " pod="openshift-must-gather-64qs2/perf-node-gather-daemonset-gfq65" Jul 06 21:40:21.720130 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.720002 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b3c3fe09-707c-4590-a7bd-6203c4d4cd00-sys\") pod \"perf-node-gather-daemonset-gfq65\" (UID: \"b3c3fe09-707c-4590-a7bd-6203c4d4cd00\") " pod="openshift-must-gather-64qs2/perf-node-gather-daemonset-gfq65" Jul 06 21:40:21.720130 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.720097 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcfv8\" (UniqueName: \"kubernetes.io/projected/b3c3fe09-707c-4590-a7bd-6203c4d4cd00-kube-api-access-lcfv8\") pod \"perf-node-gather-daemonset-gfq65\" (UID: \"b3c3fe09-707c-4590-a7bd-6203c4d4cd00\") " pod="openshift-must-gather-64qs2/perf-node-gather-daemonset-gfq65" Jul 06 21:40:21.720130 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.720125 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b3c3fe09-707c-4590-a7bd-6203c4d4cd00-podres\") pod \"perf-node-gather-daemonset-gfq65\" (UID: \"b3c3fe09-707c-4590-a7bd-6203c4d4cd00\") " pod="openshift-must-gather-64qs2/perf-node-gather-daemonset-gfq65" Jul 06 21:40:21.720336 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.720141 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b3c3fe09-707c-4590-a7bd-6203c4d4cd00-lib-modules\") pod \"perf-node-gather-daemonset-gfq65\" (UID: \"b3c3fe09-707c-4590-a7bd-6203c4d4cd00\") " pod="openshift-must-gather-64qs2/perf-node-gather-daemonset-gfq65" Jul 06 21:40:21.720336 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.720155 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b3c3fe09-707c-4590-a7bd-6203c4d4cd00-proc\") pod \"perf-node-gather-daemonset-gfq65\" (UID: \"b3c3fe09-707c-4590-a7bd-6203c4d4cd00\") " pod="openshift-must-gather-64qs2/perf-node-gather-daemonset-gfq65" Jul 06 21:40:21.729883 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.729843 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcfv8\" (UniqueName: \"kubernetes.io/projected/b3c3fe09-707c-4590-a7bd-6203c4d4cd00-kube-api-access-lcfv8\") pod \"perf-node-gather-daemonset-gfq65\" (UID: \"b3c3fe09-707c-4590-a7bd-6203c4d4cd00\") " pod="openshift-must-gather-64qs2/perf-node-gather-daemonset-gfq65" Jul 06 21:40:21.888451 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:21.888414 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-64qs2/perf-node-gather-daemonset-gfq65" Jul 06 21:40:22.031715 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:22.031673 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-64qs2/perf-node-gather-daemonset-gfq65"] Jul 06 21:40:22.035088 ip-10-0-136-207 kubenswrapper[2577]: W0706 21:40:22.035055 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb3c3fe09_707c_4590_a7bd_6203c4d4cd00.slice/crio-acb95e83f46e80235b4b5524ae856fc4da2f38b4bfb197578921f28aa83c2059 WatchSource:0}: Error finding container acb95e83f46e80235b4b5524ae856fc4da2f38b4bfb197578921f28aa83c2059: Status 404 returned error can't find the container with id acb95e83f46e80235b4b5524ae856fc4da2f38b4bfb197578921f28aa83c2059 Jul 06 21:40:22.183967 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:22.183865 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-64qs2/perf-node-gather-daemonset-gfq65" event={"ID":"b3c3fe09-707c-4590-a7bd-6203c4d4cd00","Type":"ContainerStarted","Data":"2e229c5019da7bce8409e41b864a87ae02296654e3be783de0da3272dbdaec17"} Jul 06 21:40:22.184149 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:22.184036 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-64qs2/perf-node-gather-daemonset-gfq65" Jul 06 21:40:22.184149 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:22.184084 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-64qs2/perf-node-gather-daemonset-gfq65" event={"ID":"b3c3fe09-707c-4590-a7bd-6203c4d4cd00","Type":"ContainerStarted","Data":"acb95e83f46e80235b4b5524ae856fc4da2f38b4bfb197578921f28aa83c2059"} Jul 06 21:40:22.203716 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:22.203659 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-64qs2/perf-node-gather-daemonset-gfq65" podStartSLOduration=1.203641677 podStartE2EDuration="1.203641677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 21:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 21:40:22.201233493 +0000 UTC m=+2506.519456754" watchObservedRunningTime="2026-07-06 21:40:22.203641677 +0000 UTC m=+2506.521864937" Jul 06 21:40:22.585188 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:22.585077 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-j7pvh_e7359088-7a08-466a-8bac-0cd67affc60f/download-server/0.log" Jul 06 21:40:24.474855 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:24.474826 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xj2d6_8886f6e5-67fc-4fd8-ad7b-95d98e776a1f/dns/0.log" Jul 06 21:40:24.529942 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:24.529885 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xj2d6_8886f6e5-67fc-4fd8-ad7b-95d98e776a1f/kube-rbac-proxy/0.log" Jul 06 21:40:24.591817 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:24.591781 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9mn4b_826779c0-bcd6-4c50-9ec1-0b62c9122657/dns-node-resolver/0.log" Jul 06 21:40:25.412491 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:25.412449 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dmgkn_a9511f0e-6c1f-431c-8419-c1adc71f5520/node-ca/0.log" Jul 06 21:40:26.547522 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:26.547489 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4jr2lb_980321c5-434c-4e39-ae74-2c4c5b1537b8/istio-proxy/0.log" Jul 06 21:40:26.787654 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:26.787604 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-77zzd_90be9695-5aa7-4767-95a3-81acb10d1237/istio-proxy/0.log" Jul 06 21:40:26.840041 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:26.839929 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-77zzd_90be9695-5aa7-4767-95a3-81acb10d1237/istio-proxy/1.log" Jul 06 21:40:26.902684 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:26.902650 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-nrqcv_e6c09920-f5fb-4db5-b5de-06d64bfb48f9/payload-pre-processing/0.log" Jul 06 21:40:27.002940 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:27.002893 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-sk8mt_c4dad928-112f-4baf-a79d-c9294ddadbd1/payload-processing/0.log" Jul 06 21:40:27.741519 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:27.741489 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-79v7b_09d8ee76-d883-42ab-871d-32be26a2f4f5/serve-healthcheck-canary/0.log" Jul 06 21:40:28.200063 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:28.200031 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-64qs2/perf-node-gather-daemonset-gfq65" Jul 06 21:40:28.621001 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:28.620909 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xq4l4_7013baf3-f119-4c87-9c16-a9972cb124a8/kube-rbac-proxy/0.log" Jul 06 21:40:28.661701 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:28.661673 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xq4l4_7013baf3-f119-4c87-9c16-a9972cb124a8/exporter/0.log" Jul 06 21:40:28.691561 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:28.691532 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xq4l4_7013baf3-f119-4c87-9c16-a9972cb124a8/extractor/0.log" Jul 06 21:40:31.412996 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:31.412964 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-688cc6cd9f-d78bw_013bc0d1-86de-4f5e-bce5-04c3e39798fa/maas-api/0.log" Jul 06 21:40:31.443605 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:31.443571 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722875-nmtng_ce7f9d1b-993b-44dc-9972-e708a95283d9/cleanup/0.log" Jul 06 21:40:31.472569 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:31.472538 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722890-lmd47_303aa271-951a-4bbd-aa49-33adbf5b4d28/cleanup/0.log" Jul 06 21:40:31.651474 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:31.651435 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67f746bd7b-tgt54_406c8c90-2de7-4516-b7e0-22a23bc828f2/manager/0.log" Jul 06 21:40:31.775938 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:31.775885 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bd9f97986-7ht5r_bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d/manager/0.log" Jul 06 21:40:31.800409 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:31.800381 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bd9f97986-7ht5r_bcd7ecdf-36af-47c7-8ca2-df1be45c7c1d/copy-manifests/0.log" Jul 06 21:40:31.984425 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:31.984334 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-r9pjt_fdc5952f-0f86-4a51-8678-d1887ab25ac0/postgres/0.log" Jul 06 21:40:32.375839 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:32.375804 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4jr2lb_980321c5-434c-4e39-ae74-2c4c5b1537b8/istio-proxy/0.log" Jul 06 21:40:32.532607 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:32.532577 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-77zzd_90be9695-5aa7-4767-95a3-81acb10d1237/istio-proxy/0.log" Jul 06 21:40:32.588417 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:32.588375 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-77zzd_90be9695-5aa7-4767-95a3-81acb10d1237/istio-proxy/1.log" Jul 06 21:40:32.620917 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:32.620888 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-nrqcv_e6c09920-f5fb-4db5-b5de-06d64bfb48f9/payload-pre-processing/0.log" Jul 06 21:40:32.699151 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:32.699045 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-sk8mt_c4dad928-112f-4baf-a79d-c9294ddadbd1/payload-processing/0.log" Jul 06 21:40:33.474089 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:33.474058 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6cfbcfb449-tv2zz_49ec5a9c-ddd1-4a95-9674-b02878be9d25/manager/0.log" Jul 06 21:40:33.502170 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:33.502138 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-2lwrf_631e469d-d1e5-475b-9b77-6f5d16b5a61d/openshift-lws-operator/0.log" Jul 06 21:40:33.972917 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:33.972887 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4jr2lb_980321c5-434c-4e39-ae74-2c4c5b1537b8/istio-proxy/0.log" Jul 06 21:40:34.134504 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:34.134446 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-77zzd_90be9695-5aa7-4767-95a3-81acb10d1237/istio-proxy/0.log" Jul 06 21:40:34.191163 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:34.191128 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-77zzd_90be9695-5aa7-4767-95a3-81acb10d1237/istio-proxy/1.log" Jul 06 21:40:34.224750 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:34.224659 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-nrqcv_e6c09920-f5fb-4db5-b5de-06d64bfb48f9/payload-pre-processing/0.log" Jul 06 21:40:34.295485 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:34.295455 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-sk8mt_c4dad928-112f-4baf-a79d-c9294ddadbd1/payload-processing/0.log" Jul 06 21:40:38.537306 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:38.537278 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-r9kt5_9067bb4c-c10e-49ae-bf8e-1809e833c8d9/migrator/0.log" Jul 06 21:40:38.561230 ip-10-0-136-207 kubenswrapper[2577]: I0706 21:40:38.561196 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-r9kt5_9067bb4c-c10e-49ae-bf8e-1809e833c8d9/graceful-termination/0.log"