Jul 06 11:28:21.388961 ip-10-0-131-176 systemd[1]: Starting Kubernetes Kubelet... Jul 06 11:28:21.863530 ip-10-0-131-176 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 11:28:21.863530 ip-10-0-131-176 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 11:28:21.863530 ip-10-0-131-176 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 11:28:21.863530 ip-10-0-131-176 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 11:28:21.863530 ip-10-0-131-176 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 11:28:21.863530 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.842123 2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 11:28:21.863530 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.848826 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 11:28:21.863530 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.848957 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 11:28:21.863530 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.848969 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 11:28:21.863530 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.848975 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 11:28:21.863530 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.848980 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 11:28:21.863530 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.848985 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 11:28:21.863530 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.848990 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 11:28:21.863530 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.848995 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 11:28:21.865593 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.848999 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 11:28:21.865593 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849004 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 11:28:21.865593 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849008 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 11:28:21.865593 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849013 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 11:28:21.865593 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849017 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 11:28:21.865593 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849021 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 11:28:21.865593 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849025 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 11:28:21.865593 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849029 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 11:28:21.865593 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849033 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 11:28:21.865593 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849037 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 11:28:21.865593 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849041 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 11:28:21.865593 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849045 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 11:28:21.865593 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849050 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 11:28:21.865593 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849054 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 11:28:21.865593 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849058 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 11:28:21.865593 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849062 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 11:28:21.865593 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849066 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 11:28:21.865593 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849071 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 11:28:21.865593 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849075 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 11:28:21.865593 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849079 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 11:28:21.867812 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849083 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 11:28:21.867812 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849088 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 11:28:21.867812 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849092 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 11:28:21.867812 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849097 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 11:28:21.867812 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849102 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 11:28:21.867812 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849110 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 11:28:21.867812 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849116 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 11:28:21.867812 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849120 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 11:28:21.867812 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849125 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 11:28:21.867812 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849129 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 11:28:21.867812 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849133 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 11:28:21.867812 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849137 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 11:28:21.867812 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849140 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 11:28:21.867812 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849142 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 11:28:21.867812 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849145 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 11:28:21.867812 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849148 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 11:28:21.867812 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849150 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 11:28:21.867812 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849153 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 11:28:21.867812 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849156 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 11:28:21.867812 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849159 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 11:28:21.869516 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849161 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 11:28:21.869516 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849164 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 11:28:21.869516 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849166 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 11:28:21.869516 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849169 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 11:28:21.869516 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849172 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 11:28:21.869516 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849175 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 06 11:28:21.869516 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849178 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 11:28:21.869516 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849180 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 11:28:21.869516 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849183 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 11:28:21.869516 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849186 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 11:28:21.869516 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849188 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 11:28:21.869516 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849191 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 11:28:21.869516 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849194 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 11:28:21.869516 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849197 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 11:28:21.869516 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849201 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 11:28:21.869516 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849206 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 11:28:21.869516 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849209 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 11:28:21.869516 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849212 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 11:28:21.869516 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849216 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 11:28:21.887770 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849219 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 11:28:21.887770 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849222 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 11:28:21.887770 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849224 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 11:28:21.887770 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849227 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 11:28:21.887770 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849230 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 11:28:21.887770 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849233 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 11:28:21.887770 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849236 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 11:28:21.887770 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849239 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 11:28:21.887770 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849241 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 11:28:21.887770 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849244 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 11:28:21.887770 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849247 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 11:28:21.887770 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849251 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 11:28:21.887770 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849253 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 11:28:21.887770 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849256 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 11:28:21.887770 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849259 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 11:28:21.887770 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849261 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 11:28:21.887770 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849264 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 11:28:21.887770 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849267 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 11:28:21.887770 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849269 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 11:28:21.887770 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849273 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 11:28:21.889924 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849278 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 11:28:21.889924 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849282 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 11:28:21.889924 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849285 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 11:28:21.889924 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849287 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 11:28:21.889924 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849290 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 11:28:21.889924 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849293 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 11:28:21.889924 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849296 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 11:28:21.889924 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849299 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 11:28:21.889924 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849302 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 11:28:21.889924 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849304 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 11:28:21.889924 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849307 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 11:28:21.889924 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.849310 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 11:28:21.889924 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850761 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 11:28:21.889924 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850772 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 11:28:21.889924 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850775 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 11:28:21.889924 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850778 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 11:28:21.889924 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850781 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 11:28:21.889924 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850784 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 11:28:21.889924 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850787 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 11:28:21.889924 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850790 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 11:28:21.892490 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850793 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 11:28:21.892490 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850796 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 11:28:21.892490 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850799 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 11:28:21.892490 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850802 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 11:28:21.892490 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850804 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 11:28:21.892490 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850807 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 11:28:21.892490 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850810 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 11:28:21.892490 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850812 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 11:28:21.892490 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850816 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 11:28:21.892490 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850820 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 11:28:21.892490 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850823 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 11:28:21.892490 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850825 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 11:28:21.892490 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850828 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 11:28:21.892490 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850831 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 11:28:21.892490 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850834 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 11:28:21.892490 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850836 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 11:28:21.892490 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850839 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 11:28:21.892490 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850842 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 11:28:21.892490 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850844 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 11:28:21.892490 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850847 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 11:28:21.894571 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850850 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 11:28:21.894571 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850853 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 11:28:21.894571 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850856 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 11:28:21.894571 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850859 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 11:28:21.894571 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850862 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 11:28:21.894571 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850864 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 11:28:21.894571 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850867 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 11:28:21.894571 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850870 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 11:28:21.894571 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850873 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 11:28:21.894571 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850876 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 11:28:21.894571 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850878 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 11:28:21.894571 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850881 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 11:28:21.894571 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850884 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 11:28:21.894571 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850887 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 11:28:21.894571 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850889 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 11:28:21.894571 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850892 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 11:28:21.894571 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850895 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 11:28:21.894571 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850897 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 11:28:21.894571 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850900 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 11:28:21.894571 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850902 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 11:28:21.896851 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850905 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 11:28:21.896851 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850907 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 11:28:21.896851 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850911 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 11:28:21.896851 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850914 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 11:28:21.896851 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850916 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 11:28:21.896851 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850919 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 11:28:21.896851 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850922 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 11:28:21.896851 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850925 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 11:28:21.896851 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850928 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 11:28:21.896851 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850931 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 11:28:21.896851 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850934 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 11:28:21.896851 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850937 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 06 11:28:21.896851 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850939 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 11:28:21.896851 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850942 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 11:28:21.896851 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850948 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 11:28:21.896851 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850952 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 11:28:21.896851 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850955 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 11:28:21.896851 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850958 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 11:28:21.896851 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850962 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 11:28:21.911291 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850966 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 11:28:21.911291 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850970 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 11:28:21.911291 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850973 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 11:28:21.911291 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850976 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 11:28:21.911291 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850979 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 11:28:21.911291 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850982 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 11:28:21.911291 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850986 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 11:28:21.911291 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850988 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 11:28:21.911291 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850991 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 11:28:21.911291 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850994 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 11:28:21.911291 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850997 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 11:28:21.911291 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.850999 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 11:28:21.911291 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851004 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 11:28:21.911291 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851008 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 11:28:21.911291 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851011 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 11:28:21.911291 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851014 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 11:28:21.911291 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851018 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 11:28:21.911291 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851022 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 11:28:21.911291 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851025 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 11:28:21.913693 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851027 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 11:28:21.913693 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851030 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 11:28:21.913693 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851033 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 11:28:21.913693 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851036 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 11:28:21.913693 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851038 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 11:28:21.913693 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851041 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 11:28:21.913693 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851043 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 11:28:21.913693 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851046 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 11:28:21.913693 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851050 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 11:28:21.913693 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851053 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 11:28:21.913693 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851055 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 11:28:21.913693 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851058 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 11:28:21.913693 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851061 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 11:28:21.913693 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851469 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 11:28:21.913693 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851475 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 11:28:21.913693 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851478 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 11:28:21.913693 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851481 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 11:28:21.913693 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851484 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 11:28:21.913693 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851487 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 11:28:21.913693 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851489 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 11:28:21.915643 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851492 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 11:28:21.915643 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851495 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 11:28:21.915643 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851497 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 11:28:21.915643 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851501 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 11:28:21.915643 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851505 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 11:28:21.915643 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851508 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 11:28:21.915643 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851510 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 11:28:21.915643 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851513 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 11:28:21.915643 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851516 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 11:28:21.915643 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851519 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 11:28:21.915643 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851523 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 11:28:21.915643 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851526 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 11:28:21.915643 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851529 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 11:28:21.915643 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851531 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 11:28:21.915643 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851534 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 11:28:21.915643 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851536 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 11:28:21.915643 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851539 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 11:28:21.915643 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851542 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 11:28:21.915643 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851544 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 11:28:21.917867 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851547 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 11:28:21.917867 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851551 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 11:28:21.917867 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851553 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 11:28:21.917867 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851556 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 11:28:21.917867 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851559 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 11:28:21.917867 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851561 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 06 11:28:21.917867 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851564 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 11:28:21.917867 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851566 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 11:28:21.917867 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851569 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 11:28:21.917867 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851572 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 11:28:21.917867 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851574 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 11:28:21.917867 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851578 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 11:28:21.917867 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851582 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 11:28:21.917867 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851585 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 11:28:21.917867 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851587 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 11:28:21.917867 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851590 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 11:28:21.917867 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851593 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 11:28:21.917867 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851596 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 11:28:21.917867 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851598 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 11:28:21.917867 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851600 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 11:28:21.931343 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851603 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 11:28:21.931343 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851606 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 11:28:21.931343 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851608 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 11:28:21.931343 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851613 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 11:28:21.931343 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851616 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 11:28:21.931343 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851618 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 11:28:21.931343 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851621 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 11:28:21.931343 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851624 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 11:28:21.931343 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851626 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 11:28:21.931343 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851629 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 11:28:21.931343 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851631 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 11:28:21.931343 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851633 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 11:28:21.931343 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851636 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 11:28:21.931343 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851640 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 11:28:21.931343 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851642 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 11:28:21.931343 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851645 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 11:28:21.931343 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851647 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 11:28:21.931343 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851650 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 11:28:21.931343 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851652 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 11:28:21.931343 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851655 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 11:28:21.934026 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851657 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 11:28:21.934026 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851660 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 11:28:21.934026 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851663 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 11:28:21.934026 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851665 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 11:28:21.934026 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851667 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 11:28:21.934026 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851670 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 11:28:21.934026 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851690 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 11:28:21.934026 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851694 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 11:28:21.934026 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851696 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 11:28:21.934026 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851699 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 11:28:21.934026 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851702 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 11:28:21.934026 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851704 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 11:28:21.934026 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851707 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 11:28:21.934026 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851709 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 11:28:21.934026 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851712 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 11:28:21.934026 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851716 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 11:28:21.934026 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851720 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 11:28:21.934026 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851723 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 11:28:21.934026 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851726 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 11:28:21.927893 ip-10-0-131-176 systemd[1]: Started Kubernetes Kubelet. Jul 06 11:28:21.948352 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851730 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 11:28:21.948352 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851733 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 11:28:21.948352 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851736 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 11:28:21.948352 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851739 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 11:28:21.948352 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851741 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 11:28:21.948352 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851744 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 11:28:21.948352 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851748 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 11:28:21.948352 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851750 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 11:28:21.948352 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851753 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 11:28:21.948352 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851756 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 11:28:21.948352 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851758 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 11:28:21.948352 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851761 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 11:28:21.948352 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851764 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 11:28:21.948352 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.851766 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 11:28:21.948352 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851843 2568 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 11:28:21.948352 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851853 2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 11:28:21.948352 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851864 2568 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 11:28:21.948352 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851870 2568 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 11:28:21.948352 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851876 2568 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 11:28:21.948352 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851881 2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 11:28:21.948352 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851888 2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 11:28:21.963724 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851893 2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 11:28:21.963724 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851896 2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 11:28:21.963724 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851899 2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 11:28:21.963724 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851903 2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 11:28:21.963724 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851906 2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 11:28:21.963724 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851909 2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 11:28:21.963724 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851912 2568 flags.go:64] FLAG: --cgroup-root="" Jul 06 11:28:21.963724 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851917 2568 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 11:28:21.963724 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851920 2568 flags.go:64] FLAG: --client-ca-file="" Jul 06 11:28:21.963724 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851923 2568 flags.go:64] FLAG: --cloud-provider="external" Jul 06 11:28:21.963724 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851927 2568 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 11:28:21.963724 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851931 2568 flags.go:64] FLAG: --cluster-domain="" Jul 06 11:28:21.963724 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851933 2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 11:28:21.963724 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851937 2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 11:28:21.963724 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851940 2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 11:28:21.963724 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851943 2568 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 11:28:21.963724 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851947 2568 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 11:28:21.963724 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851952 2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 11:28:21.963724 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851956 2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 11:28:21.963724 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851959 2568 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 11:28:21.963724 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851962 2568 flags.go:64] FLAG: --contention-profiling="false" Jul 06 11:28:21.963724 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851965 2568 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 11:28:21.963724 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851969 2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 11:28:21.963724 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851972 2568 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 11:28:21.975686 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851975 2568 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 11:28:21.975686 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851979 2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 11:28:21.975686 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851982 2568 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 11:28:21.975686 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851985 2568 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 11:28:21.975686 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851988 2568 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 11:28:21.975686 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851991 2568 flags.go:64] FLAG: --enable-server="true" Jul 06 11:28:21.975686 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851994 2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 11:28:21.975686 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.851999 2568 flags.go:64] FLAG: --event-burst="100" Jul 06 11:28:21.975686 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852002 2568 flags.go:64] FLAG: --event-qps="50" Jul 06 11:28:21.975686 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852005 2568 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 11:28:21.975686 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852009 2568 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 11:28:21.975686 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852011 2568 flags.go:64] FLAG: --eviction-hard="" Jul 06 11:28:21.975686 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852015 2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 11:28:21.975686 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852018 2568 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 11:28:21.975686 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852021 2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 11:28:21.975686 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852026 2568 flags.go:64] FLAG: --eviction-soft="" Jul 06 11:28:21.975686 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852029 2568 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 11:28:21.975686 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852032 2568 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 11:28:21.975686 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852035 2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 11:28:21.975686 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852038 2568 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 11:28:21.975686 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852041 2568 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 11:28:21.975686 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852044 2568 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 11:28:21.975686 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852047 2568 flags.go:64] FLAG: --feature-gates="" Jul 06 11:28:21.975686 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852051 2568 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 11:28:21.975686 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852054 2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 11:28:22.118827 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852058 2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 11:28:22.118827 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852061 2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 11:28:22.118827 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852064 2568 flags.go:64] FLAG: --healthz-port="10248" Jul 06 11:28:22.118827 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852067 2568 flags.go:64] FLAG: --help="false" Jul 06 11:28:22.118827 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852069 2568 flags.go:64] FLAG: --hostname-override="ip-10-0-131-176.ec2.internal" Jul 06 11:28:22.118827 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852073 2568 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 11:28:22.118827 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852075 2568 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 11:28:22.118827 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852078 2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 11:28:22.118827 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852082 2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 11:28:22.118827 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852085 2568 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 11:28:22.118827 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852088 2568 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 11:28:22.118827 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852091 2568 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 11:28:22.118827 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852094 2568 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 11:28:22.118827 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852096 2568 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 11:28:22.118827 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852099 2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 11:28:22.118827 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852102 2568 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 11:28:22.118827 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852105 2568 flags.go:64] FLAG: --kube-reserved="" Jul 06 11:28:22.118827 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852108 2568 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 11:28:22.118827 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852110 2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 11:28:22.118827 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852113 2568 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 11:28:22.118827 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852116 2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 11:28:22.118827 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852119 2568 flags.go:64] FLAG: --lock-file="" Jul 06 11:28:22.118827 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852123 2568 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 11:28:22.118827 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852126 2568 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 11:28:22.146730 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852129 2568 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 11:28:22.146730 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852134 2568 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 11:28:22.146730 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852138 2568 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 11:28:22.146730 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852141 2568 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 11:28:22.146730 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852143 2568 flags.go:64] FLAG: --logging-format="text" Jul 06 11:28:22.146730 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852146 2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 11:28:22.146730 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852149 2568 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 11:28:22.146730 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852152 2568 flags.go:64] FLAG: --manifest-url="" Jul 06 11:28:22.146730 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852156 2568 flags.go:64] FLAG: --manifest-url-header="" Jul 06 11:28:22.146730 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852160 2568 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 11:28:22.146730 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852163 2568 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 11:28:22.146730 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852167 2568 flags.go:64] FLAG: --max-pods="110" Jul 06 11:28:22.146730 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852170 2568 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 11:28:22.146730 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852173 2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 11:28:22.146730 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852176 2568 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 11:28:22.146730 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852179 2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 11:28:22.146730 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852182 2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 11:28:22.146730 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852185 2568 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 11:28:22.146730 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852188 2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 11:28:22.146730 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852195 2568 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 11:28:22.146730 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852198 2568 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 11:28:22.146730 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852201 2568 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 11:28:22.146730 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852204 2568 flags.go:64] FLAG: --pod-cidr="" Jul 06 11:28:22.148028 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852207 2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 11:28:22.148028 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852212 2568 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 11:28:22.148028 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852215 2568 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 11:28:22.148028 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852218 2568 flags.go:64] FLAG: --pods-per-core="0" Jul 06 11:28:22.148028 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852221 2568 flags.go:64] FLAG: --port="10250" Jul 06 11:28:22.148028 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852223 2568 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 11:28:22.148028 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852226 2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-014bceaf36730dffd" Jul 06 11:28:22.148028 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852231 2568 flags.go:64] FLAG: --qos-reserved="" Jul 06 11:28:22.148028 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852234 2568 flags.go:64] FLAG: --read-only-port="10255" Jul 06 11:28:22.148028 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852237 2568 flags.go:64] FLAG: --register-node="true" Jul 06 11:28:22.148028 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852240 2568 flags.go:64] FLAG: --register-with-taints="" Jul 06 11:28:22.148028 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852244 2568 flags.go:64] FLAG: --registry-burst="10" Jul 06 11:28:22.148028 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852247 2568 flags.go:64] FLAG: --registry-qps="5" Jul 06 11:28:22.148028 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852250 2568 flags.go:64] FLAG: --reserved-cpus="" Jul 06 11:28:22.148028 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852253 2568 flags.go:64] FLAG: --reserved-memory="" Jul 06 11:28:22.148028 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852256 2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 11:28:22.148028 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852259 2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 11:28:22.148028 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852262 2568 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 11:28:22.148028 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852266 2568 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 11:28:22.148028 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852269 2568 flags.go:64] FLAG: --runonce="false" Jul 06 11:28:22.148028 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852272 2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 11:28:22.148028 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852275 2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 11:28:22.148028 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852278 2568 flags.go:64] FLAG: --seccomp-default="false" Jul 06 11:28:22.148028 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852281 2568 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 11:28:22.148028 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852284 2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 11:28:22.152622 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852287 2568 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 11:28:22.152622 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852290 2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 11:28:22.152622 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852293 2568 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 11:28:22.152622 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852296 2568 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 11:28:22.152622 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852299 2568 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 11:28:22.152622 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852302 2568 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 11:28:22.152622 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852305 2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 11:28:22.152622 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852308 2568 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 11:28:22.152622 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852311 2568 flags.go:64] FLAG: --system-cgroups="" Jul 06 11:28:22.152622 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852313 2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 11:28:22.152622 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852318 2568 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 11:28:22.152622 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852321 2568 flags.go:64] FLAG: --tls-cert-file="" Jul 06 11:28:22.152622 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852324 2568 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 11:28:22.152622 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852328 2568 flags.go:64] FLAG: --tls-min-version="" Jul 06 11:28:22.152622 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852332 2568 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 11:28:22.152622 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852335 2568 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 11:28:22.152622 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852338 2568 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 11:28:22.152622 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852341 2568 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 11:28:22.152622 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852343 2568 flags.go:64] FLAG: --v="2" Jul 06 11:28:22.152622 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852353 2568 flags.go:64] FLAG: --version="false" Jul 06 11:28:22.152622 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852357 2568 flags.go:64] FLAG: --vmodule="" Jul 06 11:28:22.152622 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852361 2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 11:28:22.152622 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.852365 2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 11:28:22.152622 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852478 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 11:28:22.152622 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852482 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 11:28:22.153450 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852487 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 11:28:22.153450 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852490 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 11:28:22.153450 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852493 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 11:28:22.153450 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852496 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 11:28:22.153450 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852499 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 11:28:22.153450 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852501 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 11:28:22.153450 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852504 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 11:28:22.153450 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852507 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 11:28:22.153450 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852509 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 11:28:22.153450 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852512 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 11:28:22.153450 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852514 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 11:28:22.153450 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852517 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 11:28:22.153450 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852520 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 11:28:22.153450 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852522 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 11:28:22.153450 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852525 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 11:28:22.153450 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852528 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 11:28:22.153450 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852530 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 11:28:22.153450 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852533 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 11:28:22.153450 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852536 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 11:28:22.154075 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852539 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 11:28:22.154075 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852542 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 11:28:22.154075 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852547 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 11:28:22.154075 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852550 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 11:28:22.154075 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852552 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 06 11:28:22.154075 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852555 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 11:28:22.154075 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852557 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 11:28:22.154075 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852560 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 11:28:22.154075 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852562 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 11:28:22.154075 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852566 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 11:28:22.154075 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852568 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 11:28:22.154075 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852571 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 11:28:22.154075 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852573 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 11:28:22.154075 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852577 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 11:28:22.154075 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852579 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 11:28:22.154075 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852584 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 11:28:22.154075 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852587 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 11:28:22.154075 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852590 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 11:28:22.154075 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852593 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 11:28:22.154075 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852596 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 11:28:22.154784 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852599 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 11:28:22.154784 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852602 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 11:28:22.154784 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852604 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 11:28:22.154784 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852608 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 11:28:22.154784 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852610 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 11:28:22.154784 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852613 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 11:28:22.154784 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852617 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 11:28:22.154784 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852622 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 11:28:22.154784 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852624 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 11:28:22.154784 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852627 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 11:28:22.154784 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852629 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 11:28:22.154784 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852632 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 11:28:22.154784 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852635 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 11:28:22.154784 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852638 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 11:28:22.154784 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852642 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 11:28:22.154784 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852644 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 11:28:22.154784 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852647 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 11:28:22.154784 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852650 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 11:28:22.154784 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852652 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 11:28:22.154784 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852655 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 11:28:22.155419 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852658 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 11:28:22.155419 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852660 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 11:28:22.155419 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852666 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 11:28:22.155419 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852669 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 11:28:22.155419 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852671 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 11:28:22.155419 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852689 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 11:28:22.155419 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852692 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 11:28:22.155419 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852695 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 11:28:22.155419 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852697 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 11:28:22.155419 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852700 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 11:28:22.155419 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852703 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 11:28:22.155419 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852705 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 11:28:22.155419 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852708 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 11:28:22.155419 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852711 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 11:28:22.155419 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852714 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 11:28:22.155419 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852716 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 11:28:22.155419 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852719 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 11:28:22.155419 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852721 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 11:28:22.155419 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852724 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 11:28:22.155419 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852726 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 11:28:22.156067 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852730 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 11:28:22.156067 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852734 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 11:28:22.156067 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852737 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 11:28:22.156067 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852739 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 11:28:22.156067 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852742 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 11:28:22.156067 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852745 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 11:28:22.156067 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852750 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 11:28:22.156067 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852752 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 11:28:22.156067 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852755 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 11:28:22.156067 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852758 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 11:28:22.156067 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852761 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 11:28:22.156067 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852764 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 11:28:22.156067 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852767 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 11:28:22.156067 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852769 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 11:28:22.156067 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852772 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 11:28:22.156067 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852775 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 11:28:22.156067 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852777 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 11:28:22.156067 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.852781 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 11:28:22.156602 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.853547 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 11:28:22.156602 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.860963 2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 11:28:22.156602 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.860979 2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 11:28:22.156602 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.861000 2568 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 11:28:22.156602 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.861006 2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 11:28:22.156602 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861062 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 11:28:22.156602 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861067 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 11:28:22.156602 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861070 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 11:28:22.156602 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861073 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 11:28:22.156602 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861076 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 11:28:22.156602 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861079 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 11:28:22.156602 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861082 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 11:28:22.156602 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861085 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 11:28:22.156602 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861088 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 11:28:22.156602 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861091 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 11:28:22.157284 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861094 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 11:28:22.157284 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861098 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 11:28:22.157284 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861102 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 11:28:22.157284 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861105 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 11:28:22.157284 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861108 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 11:28:22.157284 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861111 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 11:28:22.157284 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861114 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 11:28:22.157284 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861117 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 11:28:22.157284 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861120 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 11:28:22.157284 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861123 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 11:28:22.157284 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861126 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 11:28:22.157284 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861130 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 11:28:22.157284 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861132 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 11:28:22.157284 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861135 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 11:28:22.157284 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861140 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 11:28:22.157284 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861144 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 11:28:22.157284 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861147 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 11:28:22.157284 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861150 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 11:28:22.157284 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861153 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 11:28:22.157952 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861157 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 11:28:22.157952 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861160 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 11:28:22.157952 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861163 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 11:28:22.157952 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861169 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 11:28:22.157952 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861172 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 11:28:22.157952 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861175 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 11:28:22.157952 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861177 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 11:28:22.157952 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861180 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 11:28:22.157952 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861183 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 11:28:22.157952 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861185 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 11:28:22.157952 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861188 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 11:28:22.157952 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861191 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 11:28:22.157952 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861194 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 11:28:22.157952 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861196 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 11:28:22.157952 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861200 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 11:28:22.157952 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861202 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 11:28:22.157952 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861205 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 11:28:22.157952 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861208 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 11:28:22.157952 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861210 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 11:28:22.158610 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861213 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 11:28:22.158610 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861216 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 11:28:22.158610 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861219 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 11:28:22.158610 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861221 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 11:28:22.158610 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861224 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 11:28:22.158610 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861226 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 11:28:22.158610 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861229 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 11:28:22.158610 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861231 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 11:28:22.158610 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861234 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 11:28:22.158610 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861237 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 11:28:22.158610 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861239 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 11:28:22.158610 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861243 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 11:28:22.158610 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861245 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 11:28:22.158610 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861249 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 11:28:22.158610 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861253 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 11:28:22.158610 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861256 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 11:28:22.158610 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861260 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 11:28:22.158610 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861262 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 11:28:22.158610 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861265 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 11:28:22.158610 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861267 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 11:28:22.389486 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861270 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 11:28:22.389486 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861273 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 11:28:22.389486 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861275 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 11:28:22.389486 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861278 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 11:28:22.389486 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861281 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 11:28:22.389486 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861284 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 11:28:22.389486 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861286 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 11:28:22.389486 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861289 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 11:28:22.389486 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861292 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 11:28:22.389486 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861294 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 11:28:22.389486 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861297 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 11:28:22.389486 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861299 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 11:28:22.389486 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861302 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 11:28:22.389486 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861304 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 11:28:22.389486 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861307 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 11:28:22.389486 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861309 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 11:28:22.389486 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861312 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 11:28:22.389486 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861314 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 11:28:22.389486 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861317 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 11:28:22.389486 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861320 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 11:28:22.390280 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861322 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 11:28:22.390280 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861325 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 11:28:22.390280 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861328 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 11:28:22.390280 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861331 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 11:28:22.390280 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861334 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 06 11:28:22.390280 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861337 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 11:28:22.390280 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861341 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 11:28:22.390280 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861345 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 11:28:22.390280 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861350 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 11:28:22.390280 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861353 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 11:28:22.390280 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861356 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 11:28:22.390280 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.861361 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 11:28:22.390280 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861472 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 11:28:22.390280 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861477 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 11:28:22.390280 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861480 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 11:28:22.390788 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861483 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 11:28:22.390788 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861486 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 11:28:22.390788 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861488 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 11:28:22.390788 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861491 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 11:28:22.390788 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861494 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 11:28:22.390788 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861496 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 11:28:22.390788 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861499 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 11:28:22.390788 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861502 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 11:28:22.390788 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861504 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 11:28:22.390788 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861507 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 11:28:22.390788 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861509 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 11:28:22.390788 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861512 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 11:28:22.390788 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861515 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 11:28:22.390788 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861517 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 11:28:22.390788 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861520 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 11:28:22.390788 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861523 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 11:28:22.390788 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861525 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 11:28:22.390788 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861528 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 11:28:22.390788 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861531 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 11:28:22.390788 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861533 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 11:28:22.391377 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861536 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 11:28:22.391377 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861540 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 11:28:22.391377 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861544 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 11:28:22.391377 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861548 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 11:28:22.391377 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861552 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 11:28:22.391377 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861555 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 11:28:22.391377 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861560 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 11:28:22.391377 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861563 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 11:28:22.391377 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861566 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 11:28:22.391377 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861568 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 11:28:22.391377 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861571 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 11:28:22.391377 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861573 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 11:28:22.391377 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861576 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 11:28:22.391377 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861579 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 11:28:22.391377 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861581 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 11:28:22.391377 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861584 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 11:28:22.391377 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861586 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 11:28:22.391377 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861589 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 11:28:22.391377 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861592 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 11:28:22.391903 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861594 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 11:28:22.391903 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861597 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 11:28:22.391903 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861599 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 11:28:22.391903 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861602 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 11:28:22.391903 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861604 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 11:28:22.391903 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861607 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 11:28:22.391903 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861610 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 11:28:22.391903 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861612 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 11:28:22.391903 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861615 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 11:28:22.391903 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861617 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 11:28:22.391903 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861620 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 11:28:22.391903 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861622 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 11:28:22.391903 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861625 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 11:28:22.391903 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861628 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 11:28:22.391903 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861631 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 11:28:22.391903 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861634 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 11:28:22.391903 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861638 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 11:28:22.391903 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861640 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 11:28:22.391903 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861643 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 11:28:22.391903 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861647 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 11:28:22.392469 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861649 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 11:28:22.392469 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861653 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 11:28:22.392469 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861657 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 11:28:22.392469 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861660 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 11:28:22.392469 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861663 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 11:28:22.392469 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861665 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 11:28:22.392469 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861668 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 11:28:22.392469 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861671 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 11:28:22.392469 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861691 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 06 11:28:22.392469 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861695 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 11:28:22.392469 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861699 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 11:28:22.392469 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861703 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 11:28:22.392469 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861707 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 11:28:22.392469 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861711 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 11:28:22.392469 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861713 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 11:28:22.392469 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861716 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 11:28:22.392469 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861719 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 11:28:22.392469 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861721 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 11:28:22.392469 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861724 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 11:28:22.393003 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861727 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 11:28:22.393003 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861729 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 11:28:22.393003 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861732 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 11:28:22.393003 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861734 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 11:28:22.393003 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861737 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 11:28:22.393003 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861740 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 11:28:22.393003 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861743 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 11:28:22.393003 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861745 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 11:28:22.393003 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861749 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 11:28:22.393003 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861753 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 11:28:22.393003 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861755 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 11:28:22.393003 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861758 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 11:28:22.393003 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861762 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 11:28:22.393003 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861764 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 11:28:22.393003 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861767 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 11:28:22.393003 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861770 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 11:28:22.393003 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861772 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 11:28:22.393003 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:21.861775 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 11:28:22.393531 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.861780 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 11:28:22.393531 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.861891 2568 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 11:28:22.393531 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.865378 2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 11:28:22.393531 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.867109 2568 server.go:1019] "Starting client certificate rotation" Jul 06 11:28:22.393531 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.867208 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 11:28:22.393531 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.867240 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 11:28:22.393531 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.869861 2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 11:28:22.393531 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.873957 2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 11:28:22.393531 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.889918 2568 log.go:25] "Validated CRI v1 runtime API" Jul 06 11:28:22.393531 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.892549 2568 log.go:25] "Validated CRI v1 image API" Jul 06 11:28:22.393531 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.893552 2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 11:28:22.393531 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.897727 2568 fs.go:135] Filesystem UUIDs: map[28fb295f-956c-4c0f-8f87-51b724639974:/dev/nvme0n1p4 73b976b9-e188-40b3-947b-500ba7827065:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 06 11:28:22.393906 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.897745 2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 11:28:22.394013 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.901996 2568 manager.go:216] Machine: {Timestamp:2026-07-06 11:28:21.901498627 +0000 UTC m=+0.381297634 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099517 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fc6c9887b2fc5ab8e18653c1c3467 SystemUUID:ec2fc6c9-887b-2fc5-ab8e-18653c1c3467 BootID:13665d4b-6a5d-46cc-9cad-2f541d8dbf44 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:01:93:a0:2c:b3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:01:93:a0:2c:b3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:aa:b4:88:50:83 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 11:28:22.394013 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.902110 2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 11:28:22.394013 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.902192 2568 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 11:28:22.394013 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.903142 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 11:28:22.394013 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.905993 2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 11:28:22.394013 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.906026 2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-176.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 11:28:22.394013 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.906869 2568 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.906881 2568 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.906889 2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.906904 2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.908777 2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.913933 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.914133 2568 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.914171 2568 kubelet.go:486] "Attempting to sync node with API server" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.914182 2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.914199 2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.914209 2568 kubelet.go:398] "Adding apiserver pod source" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.914220 2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.914599 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.914615 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.917083 2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.919747 2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.919775 2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.921977 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.921992 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.921998 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.922004 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.922013 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.922019 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.922027 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.922032 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.922053 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.922059 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.922068 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.922074 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.923713 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.923729 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.925432 2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-176.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:21.925429 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:21.925489 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-176.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.926937 2568 server.go:1268] "Started kubelet" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.927023 2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.928006 2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.928083 2568 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.929125 2568 server.go:310] "Adding debug handlers to kubelet server" Jul 06 11:28:22.394558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.933017 2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:21.936823 2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.941960 2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.941970 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:21.942105 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-176.ec2.internal\" not found" Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.942811 2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.942812 2568 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.942902 2568 factory.go:55] Registering systemd factory Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.942918 2568 factory.go:223] Registration of the systemd container factory successfully Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.942907 2568 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.943075 2568 reconstruct.go:97] "Volume reconstruction finished" Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.943087 2568 reconciler.go:29] "Reconciler: start to sync state" Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.943184 2568 factory.go:153] Registering CRI-O factory Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.943194 2568 factory.go:223] Registration of the crio container factory successfully Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.943242 2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.943271 2568 factory.go:103] Registering Raw factory Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.943288 2568 manager.go:1196] Started watching for new ooms in manager Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.943739 2568 manager.go:349] Starting recovery of all containers Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:21.946101 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-176.ec2.internal.18bfafd3dc0e7f44 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-176.ec2.internal,UID:ip-10-0-131-176.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-176.ec2.internal,},FirstTimestamp:2026-07-06 11:28:21.92690362 +0000 UTC m=+0.406702628,LastTimestamp:2026-07-06 11:28:21.92690362 +0000 UTC m=+0.406702628,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-176.ec2.internal,}" Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.952328 2568 manager.go:354] Recovery completed Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:21.952499 2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-176.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:21.953146 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.958596 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.960314 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-176.ec2.internal" event="NodeHasSufficientMemory" Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.960347 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-176.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.960360 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-176.ec2.internal" event="NodeHasSufficientPID" Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.960903 2568 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.960909 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.960913 2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.960927 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.961615 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-176.ec2.internal" event="NodeHasSufficientMemory" Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.961652 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-176.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 11:28:22.395638 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.961669 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-176.ec2.internal" event="NodeHasSufficientPID" Jul 06 11:28:22.624505 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:21.961955 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-176.ec2.internal.18bfafd3de0c9412 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-176.ec2.internal,UID:ip-10-0-131-176.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-176.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-176.ec2.internal,},FirstTimestamp:2026-07-06 11:28:21.960332306 +0000 UTC m=+0.440131313,LastTimestamp:2026-07-06 11:28:21.960332306 +0000 UTC m=+0.440131313,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-176.ec2.internal,}" Jul 06 11:28:22.624505 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.963100 2568 policy_none.go:49] "None policy: Start" Jul 06 11:28:22.624505 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.963114 2568 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 11:28:22.624505 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.963127 2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 11:28:22.624505 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.965562 2568 policy_none.go:47] "Start" Jul 06 11:28:22.624505 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.967567 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-slcsn" Jul 06 11:28:22.624505 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:21.973837 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-176.ec2.internal.18bfafd3de0ce767 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-176.ec2.internal,UID:ip-10-0-131-176.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-176.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-176.ec2.internal,},FirstTimestamp:2026-07-06 11:28:21.960353639 +0000 UTC m=+0.440152649,LastTimestamp:2026-07-06 11:28:21.960353639 +0000 UTC m=+0.440152649,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-176.ec2.internal,}" Jul 06 11:28:22.624505 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.974110 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-slcsn" Jul 06 11:28:22.624505 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.995325 2568 manager.go:337] "Starting Device Plugin manager" Jul 06 11:28:22.624505 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:21.995596 2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 11:28:22.624505 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.995612 2568 server.go:87] "Starting device plugin registration server" Jul 06 11:28:22.624505 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.995872 2568 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 11:28:22.624505 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.995883 2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 11:28:22.624505 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.996138 2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 11:28:22.624505 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.996232 2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 11:28:22.624505 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:21.996242 2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 11:28:22.624505 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:21.996784 2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 11:28:22.624505 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:21.996839 2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-176.ec2.internal\" not found" Jul 06 11:28:22.624505 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.044252 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 11:28:22.624505 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.045430 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 11:28:22.624505 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.045487 2568 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 11:28:22.624505 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.045516 2568 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 11:28:22.624505 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:22.045558 2568 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 11:28:22.624505 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.047828 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 11:28:22.624505 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.096528 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 11:28:22.624505 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.097532 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-176.ec2.internal" event="NodeHasSufficientMemory" Jul 06 11:28:22.624505 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.097567 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-176.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 11:28:22.626490 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.097581 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-176.ec2.internal" event="NodeHasSufficientPID" Jul 06 11:28:22.626490 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.097605 2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-176.ec2.internal" Jul 06 11:28:22.626490 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.109503 2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-176.ec2.internal" Jul 06 11:28:22.626490 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:22.109527 2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-176.ec2.internal\": node \"ip-10-0-131-176.ec2.internal\" not found" Jul 06 11:28:22.626490 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:22.136060 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-176.ec2.internal\" not found" Jul 06 11:28:22.626490 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.152284 2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-176.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-176.ec2.internal"] Jul 06 11:28:22.626490 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.152356 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 11:28:22.626490 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.153925 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-176.ec2.internal" event="NodeHasSufficientMemory" Jul 06 11:28:22.626490 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.153952 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-176.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 11:28:22.626490 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.153963 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-176.ec2.internal" event="NodeHasSufficientPID" Jul 06 11:28:22.626490 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.156150 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 11:28:22.626490 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.156261 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-176.ec2.internal" Jul 06 11:28:22.626490 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.156300 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 11:28:22.626490 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.156869 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-176.ec2.internal" event="NodeHasSufficientMemory" Jul 06 11:28:22.626490 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.156875 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-176.ec2.internal" event="NodeHasSufficientMemory" Jul 06 11:28:22.626490 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.156893 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-176.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 11:28:22.626490 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.156898 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-176.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 11:28:22.626490 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.156908 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-176.ec2.internal" event="NodeHasSufficientPID" Jul 06 11:28:22.626490 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.156908 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-176.ec2.internal" event="NodeHasSufficientPID" Jul 06 11:28:22.626490 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.159145 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-176.ec2.internal" Jul 06 11:28:22.626490 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.159174 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 11:28:22.626490 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.160013 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-176.ec2.internal" event="NodeHasSufficientMemory" Jul 06 11:28:22.626490 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.160038 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-176.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 11:28:22.626490 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.160047 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-176.ec2.internal" event="NodeHasSufficientPID" Jul 06 11:28:22.626490 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:22.190427 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-176.ec2.internal\" not found" node="ip-10-0-131-176.ec2.internal" Jul 06 11:28:22.626490 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:22.195145 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-176.ec2.internal\" not found" node="ip-10-0-131-176.ec2.internal" Jul 06 11:28:22.626490 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:22.236656 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-176.ec2.internal\" not found" Jul 06 11:28:22.626490 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.253093 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb7589110c767d3053aba8654c931fec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-176.ec2.internal\" (UID: \"bb7589110c767d3053aba8654c931fec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-176.ec2.internal" Jul 06 11:28:22.627417 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.253117 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb7589110c767d3053aba8654c931fec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-176.ec2.internal\" (UID: \"bb7589110c767d3053aba8654c931fec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-176.ec2.internal" Jul 06 11:28:22.627417 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.253136 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf4e10fd22892d39b2d489b025b34f3d-config\") pod \"kube-apiserver-proxy-ip-10-0-131-176.ec2.internal\" (UID: \"cf4e10fd22892d39b2d489b025b34f3d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-176.ec2.internal" Jul 06 11:28:22.627417 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:22.337095 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-176.ec2.internal\" not found" Jul 06 11:28:22.627417 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.353513 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf4e10fd22892d39b2d489b025b34f3d-config\") pod \"kube-apiserver-proxy-ip-10-0-131-176.ec2.internal\" (UID: \"cf4e10fd22892d39b2d489b025b34f3d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-176.ec2.internal" Jul 06 11:28:22.627417 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.353539 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb7589110c767d3053aba8654c931fec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-176.ec2.internal\" (UID: \"bb7589110c767d3053aba8654c931fec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-176.ec2.internal" Jul 06 11:28:22.627417 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.353564 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb7589110c767d3053aba8654c931fec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-176.ec2.internal\" (UID: \"bb7589110c767d3053aba8654c931fec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-176.ec2.internal" Jul 06 11:28:22.627417 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.353613 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb7589110c767d3053aba8654c931fec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-176.ec2.internal\" (UID: \"bb7589110c767d3053aba8654c931fec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-176.ec2.internal" Jul 06 11:28:22.627417 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.353634 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf4e10fd22892d39b2d489b025b34f3d-config\") pod \"kube-apiserver-proxy-ip-10-0-131-176.ec2.internal\" (UID: \"cf4e10fd22892d39b2d489b025b34f3d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-176.ec2.internal" Jul 06 11:28:22.627417 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.353660 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb7589110c767d3053aba8654c931fec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-176.ec2.internal\" (UID: \"bb7589110c767d3053aba8654c931fec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-176.ec2.internal" Jul 06 11:28:22.627417 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:22.437970 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-176.ec2.internal\" not found" Jul 06 11:28:22.627417 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.494440 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-176.ec2.internal" Jul 06 11:28:22.627417 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.499438 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-176.ec2.internal" Jul 06 11:28:22.627417 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:22.538254 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-176.ec2.internal\" not found" Jul 06 11:28:22.836733 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:22.643925 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-176.ec2.internal\" not found" Jul 06 11:28:22.836733 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:22.745194 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-176.ec2.internal\" not found" Jul 06 11:28:23.009389 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:22.846088 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-176.ec2.internal\" not found" Jul 06 11:28:23.009389 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.866435 2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 11:28:23.009389 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.866644 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 11:28:23.009389 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:22.946732 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-176.ec2.internal\" not found" Jul 06 11:28:23.009389 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.978621 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 11:23:22 +0000 UTC" deadline="2028-04-18 00:16:22.523517088 +0000 UTC" Jul 06 11:28:23.009389 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:22.978674 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15636h47m59.544847528s" Jul 06 11:28:23.201684 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:23.046825 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-176.ec2.internal\" not found" Jul 06 11:28:23.201684 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:23.107705 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf4e10fd22892d39b2d489b025b34f3d.slice/crio-254fadfea87a572dbabebeb62c203b9c325e20db4f1649b4e00b58ad0685dd27 WatchSource:0}: Error finding container 254fadfea87a572dbabebeb62c203b9c325e20db4f1649b4e00b58ad0685dd27: Status 404 returned error can't find the container with id 254fadfea87a572dbabebeb62c203b9c325e20db4f1649b4e00b58ad0685dd27 Jul 06 11:28:23.201684 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:23.108219 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7589110c767d3053aba8654c931fec.slice/crio-fbfb2ecaff5bd14d942d5373b5dbd19397a265b3c95ae893753a23104fff762f WatchSource:0}: Error finding container fbfb2ecaff5bd14d942d5373b5dbd19397a265b3c95ae893753a23104fff762f: Status 404 returned error can't find the container with id fbfb2ecaff5bd14d942d5373b5dbd19397a265b3c95ae893753a23104fff762f Jul 06 11:28:23.201684 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.113428 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 11:28:23.201684 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.129735 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 11:28:23.201684 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.130875 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-176.ec2.internal" event="NodeHasSufficientMemory" Jul 06 11:28:23.201684 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.130905 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-176.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 11:28:23.201684 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.130916 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-176.ec2.internal" event="NodeHasSufficientPID" Jul 06 11:28:23.201684 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:23.147581 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-176.ec2.internal\" not found" Jul 06 11:28:23.201684 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.158374 2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 11:28:23.442088 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:23.248810 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-176.ec2.internal\" not found" Jul 06 11:28:23.442088 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.304623 2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 11:28:23.442088 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.322237 2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 11:28:23.442088 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.346953 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-176.ec2.internal" Jul 06 11:28:23.442088 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.362773 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 11:28:23.442088 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.363979 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-176.ec2.internal" Jul 06 11:28:23.442088 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.376951 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 11:28:23.922075 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.915522 2568 apiserver.go:52] "Watching apiserver" Jul 06 11:28:23.999954 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.925669 2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 11:28:23.999954 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.927906 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-l89bb","kube-system/kube-apiserver-proxy-ip-10-0-131-176.ec2.internal","openshift-image-registry/node-ca-pbv8c","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-176.ec2.internal","openshift-multus/multus-pvz5d","openshift-multus/network-metrics-daemon-rgmk8","openshift-network-operator/iptables-alerter-x89qj","kube-system/global-pull-secret-syncer-9mqf5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6hb","openshift-cluster-node-tuning-operator/tuned-54f75","openshift-dns/node-resolver-dw874","openshift-multus/multus-additional-cni-plugins-25hzg","openshift-network-diagnostics/network-check-target-5dsbt","openshift-ovn-kubernetes/ovnkube-node-tn42c"] Jul 06 11:28:23.999954 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.930630 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:23.999954 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.933636 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:28:23.999954 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:23.933732 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dsbt" podUID="685a31ed-f864-4fe8-890c-73172763af66" Jul 06 11:28:23.999954 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.933956 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 11:28:23.999954 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.933978 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 11:28:23.999954 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.934253 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7kl2r\"" Jul 06 11:28:23.999954 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.937089 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pbv8c" Jul 06 11:28:23.999954 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.940080 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 11:28:23.999954 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.941016 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 11:28:23.999954 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.942375 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 11:28:23.999954 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.943450 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:23.999954 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.944280 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 11:28:23.999954 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.944512 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-b4smc\"" Jul 06 11:28:23.999954 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.946094 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pvz5d" Jul 06 11:28:23.999954 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.947494 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 11:28:23.999954 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.947621 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7gg98\"" Jul 06 11:28:23.999954 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.947818 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 11:28:23.999954 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.948484 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:28:23.999954 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:23.948567 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgmk8" podUID="6730cd7d-c083-452b-b123-e2e9f92745a1" Jul 06 11:28:23.999954 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.949043 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 11:28:23.999954 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.949072 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 11:28:23.999954 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.949124 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 11:28:23.999954 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.949154 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 11:28:24.001556 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.949291 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 11:28:24.001556 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.949411 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 11:28:24.001556 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.949564 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 11:28:24.001556 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.949664 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 11:28:24.001556 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.949726 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tc98b\"" Jul 06 11:28:24.001556 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.953329 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6hb" Jul 06 11:28:24.001556 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.955492 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x89qj" Jul 06 11:28:24.001556 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.956054 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 11:28:24.001556 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.956229 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 11:28:24.001556 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.956462 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 11:28:24.001556 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.956643 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j8v4v\"" Jul 06 11:28:24.001556 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.961345 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 11:28:24.001556 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.961513 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 11:28:24.001556 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.961601 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 11:28:24.001556 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.961781 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 11:28:24.001556 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.961811 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pbxtx\"" Jul 06 11:28:24.001556 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.962129 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:24.001556 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:23.962203 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9mqf5" podUID="9c4fa664-243d-4a56-91ce-a58f01d77768" Jul 06 11:28:24.001556 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.964088 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-run\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.001556 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.964121 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-host\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.001556 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.964142 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-etc-modprobe-d\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.001556 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.964166 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-host-run-netns\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.001556 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.964217 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-var-lib-openvswitch\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.003156 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.964264 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-host-cni-netd\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.003156 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.964302 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.003156 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.964329 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-etc-sysctl-d\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.003156 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.964358 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/733213c7-49ab-4f45-a0dd-d87abe31ef28-host\") pod \"node-ca-pbv8c\" (UID: \"733213c7-49ab-4f45-a0dd-d87abe31ef28\") " pod="openshift-image-registry/node-ca-pbv8c" Jul 06 11:28:24.003156 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.964380 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-log-socket\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.003156 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.964421 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzk6z\" (UniqueName: \"kubernetes.io/projected/685a31ed-f864-4fe8-890c-73172763af66-kube-api-access-rzk6z\") pod \"network-check-target-5dsbt\" (UID: \"685a31ed-f864-4fe8-890c-73172763af66\") " pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:28:24.003156 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.964458 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhrxq\" (UniqueName: \"kubernetes.io/projected/733213c7-49ab-4f45-a0dd-d87abe31ef28-kube-api-access-xhrxq\") pod \"node-ca-pbv8c\" (UID: \"733213c7-49ab-4f45-a0dd-d87abe31ef28\") " pod="openshift-image-registry/node-ca-pbv8c" Jul 06 11:28:24.003156 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.964487 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-host-slash\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.003156 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.964517 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-etc-openvswitch\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.003156 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.964546 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-run-openvswitch\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.003156 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.964567 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l89bb" Jul 06 11:28:24.003156 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.964570 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-node-log\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.003156 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.964593 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-etc-sysconfig\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.003156 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.964616 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-etc-kubernetes\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.004480 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.964641 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-ovnkube-script-lib\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.004480 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.964671 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-systemd-units\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.004480 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.964714 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-run-ovn\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.004480 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.964739 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-lib-modules\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.004480 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.964762 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm59n\" (UniqueName: \"kubernetes.io/projected/27e80ab9-5fa0-4a48-a102-95b38f491d4f-kube-api-access-gm59n\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.004480 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.964788 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/733213c7-49ab-4f45-a0dd-d87abe31ef28-serviceca\") pod \"node-ca-pbv8c\" (UID: \"733213c7-49ab-4f45-a0dd-d87abe31ef28\") " pod="openshift-image-registry/node-ca-pbv8c" Jul 06 11:28:24.004480 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.964813 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-host-kubelet\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.004480 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.964838 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-run-systemd\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.004480 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.964864 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-host-cni-bin\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.004480 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.964892 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-env-overrides\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.004480 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.964934 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-etc-systemd\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.004480 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.964963 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27e80ab9-5fa0-4a48-a102-95b38f491d4f-tmp\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.004480 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.965005 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-ovn-node-metrics-cert\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.004480 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.965034 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-sys\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.005170 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.965062 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-var-lib-kubelet\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.005170 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.965084 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/27e80ab9-5fa0-4a48-a102-95b38f491d4f-etc-tuned\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.005170 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.965105 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6kjh\" (UniqueName: \"kubernetes.io/projected/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-kube-api-access-j6kjh\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.005170 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.965127 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-host-run-ovn-kubernetes\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.005170 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.965148 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-ovnkube-config\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.005170 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.965171 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-etc-sysctl-conf\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.005170 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.967008 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dw874" Jul 06 11:28:24.005170 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.968481 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-srk5b\"" Jul 06 11:28:24.005170 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.968612 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 11:28:24.005170 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.968734 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 11:28:24.005170 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.969541 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zs6xs\"" Jul 06 11:28:24.005170 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.969611 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 11:28:24.005170 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.969946 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-25hzg" Jul 06 11:28:24.005170 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.970311 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 11:28:24.005170 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.972613 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 11:28:24.005170 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.972641 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 11:28:24.005170 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.972670 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h6jzn\"" Jul 06 11:28:24.005170 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.979035 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bzhmc" Jul 06 11:28:24.005170 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:23.985421 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bzhmc" Jul 06 11:28:24.078278 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.044207 2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 11:28:24.078278 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.051531 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-176.ec2.internal" event={"ID":"bb7589110c767d3053aba8654c931fec","Type":"ContainerStarted","Data":"fbfb2ecaff5bd14d942d5373b5dbd19397a265b3c95ae893753a23104fff762f"} Jul 06 11:28:24.078278 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.053421 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-176.ec2.internal" event={"ID":"cf4e10fd22892d39b2d489b025b34f3d","Type":"ContainerStarted","Data":"254fadfea87a572dbabebeb62c203b9c325e20db4f1649b4e00b58ad0685dd27"} Jul 06 11:28:24.078278 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.066089 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-host-run-multus-certs\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.078278 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.066128 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c4fa664-243d-4a56-91ce-a58f01d77768-kubelet-config\") pod \"global-pull-secret-syncer-9mqf5\" (UID: \"9c4fa664-243d-4a56-91ce-a58f01d77768\") " pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:24.078278 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.066151 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c4fa664-243d-4a56-91ce-a58f01d77768-dbus\") pod \"global-pull-secret-syncer-9mqf5\" (UID: \"9c4fa664-243d-4a56-91ce-a58f01d77768\") " pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:24.078278 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.066195 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/10439038-5c07-452f-bea0-a01a60ef437f-hosts-file\") pod \"node-resolver-dw874\" (UID: \"10439038-5c07-452f-bea0-a01a60ef437f\") " pod="openshift-dns/node-resolver-dw874" Jul 06 11:28:24.078278 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.066225 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/733213c7-49ab-4f45-a0dd-d87abe31ef28-host\") pod \"node-ca-pbv8c\" (UID: \"733213c7-49ab-4f45-a0dd-d87abe31ef28\") " pod="openshift-image-registry/node-ca-pbv8c" Jul 06 11:28:24.078278 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.066249 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-log-socket\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.078278 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.066272 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-hostroot\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.078278 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.066295 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-multus-conf-dir\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.078278 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.066318 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kplxs\" (UniqueName: \"kubernetes.io/projected/6730cd7d-c083-452b-b123-e2e9f92745a1-kube-api-access-kplxs\") pod \"network-metrics-daemon-rgmk8\" (UID: \"6730cd7d-c083-452b-b123-e2e9f92745a1\") " pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:28:24.078278 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.066343 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/218f64a8-8fd8-462b-972d-ff9ecedb6a74-konnectivity-ca\") pod \"konnectivity-agent-l89bb\" (UID: \"218f64a8-8fd8-462b-972d-ff9ecedb6a74\") " pod="kube-system/konnectivity-agent-l89bb" Jul 06 11:28:24.078278 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.066367 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c3934448-4795-4cbc-8f17-8f7e14935dd2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dh6hb\" (UID: \"c3934448-4795-4cbc-8f17-8f7e14935dd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6hb" Jul 06 11:28:24.078278 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.066392 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e3cd243e-9aaa-4671-b9b8-2f135c474488-multus-daemon-config\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.079941 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.066415 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4vnz\" (UniqueName: \"kubernetes.io/projected/8af859eb-6aa4-41b0-89cb-8028432c8f55-kube-api-access-z4vnz\") pod \"iptables-alerter-x89qj\" (UID: \"8af859eb-6aa4-41b0-89cb-8028432c8f55\") " pod="openshift-network-operator/iptables-alerter-x89qj" Jul 06 11:28:24.079941 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.066438 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-host-run-k8s-cni-cncf-io\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.079941 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.066466 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzk6z\" (UniqueName: \"kubernetes.io/projected/685a31ed-f864-4fe8-890c-73172763af66-kube-api-access-rzk6z\") pod \"network-check-target-5dsbt\" (UID: \"685a31ed-f864-4fe8-890c-73172763af66\") " pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:28:24.079941 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.066493 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhrxq\" (UniqueName: \"kubernetes.io/projected/733213c7-49ab-4f45-a0dd-d87abe31ef28-kube-api-access-xhrxq\") pod \"node-ca-pbv8c\" (UID: \"733213c7-49ab-4f45-a0dd-d87abe31ef28\") " pod="openshift-image-registry/node-ca-pbv8c" Jul 06 11:28:24.079941 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.066516 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-etc-sysconfig\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.079941 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.066547 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-etc-kubernetes\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.079941 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.066570 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-lib-modules\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.079941 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.066593 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfcd9856-b786-4cb9-b407-1fe4eafe748e-cnibin\") pod \"multus-additional-cni-plugins-25hzg\" (UID: \"cfcd9856-b786-4cb9-b407-1fe4eafe748e\") " pod="openshift-multus/multus-additional-cni-plugins-25hzg" Jul 06 11:28:24.079941 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.066615 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-multus-socket-dir-parent\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.079941 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.066640 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-ovnkube-script-lib\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.079941 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.066666 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/218f64a8-8fd8-462b-972d-ff9ecedb6a74-agent-certs\") pod \"konnectivity-agent-l89bb\" (UID: \"218f64a8-8fd8-462b-972d-ff9ecedb6a74\") " pod="kube-system/konnectivity-agent-l89bb" Jul 06 11:28:24.079941 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.066697 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/733213c7-49ab-4f45-a0dd-d87abe31ef28-serviceca\") pod \"node-ca-pbv8c\" (UID: \"733213c7-49ab-4f45-a0dd-d87abe31ef28\") " pod="openshift-image-registry/node-ca-pbv8c" Jul 06 11:28:24.079941 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.066724 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-host-kubelet\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.079941 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.066865 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-env-overrides\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.080920 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.067088 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-lib-modules\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.080920 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.067162 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-log-socket\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.080920 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.067269 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-host-kubelet\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.080920 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.067377 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/733213c7-49ab-4f45-a0dd-d87abe31ef28-serviceca\") pod \"node-ca-pbv8c\" (UID: \"733213c7-49ab-4f45-a0dd-d87abe31ef28\") " pod="openshift-image-registry/node-ca-pbv8c" Jul 06 11:28:24.080920 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.067444 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-env-overrides\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.080920 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.067482 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-etc-sysconfig\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.080920 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.067480 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/733213c7-49ab-4f45-a0dd-d87abe31ef28-host\") pod \"node-ca-pbv8c\" (UID: \"733213c7-49ab-4f45-a0dd-d87abe31ef28\") " pod="openshift-image-registry/node-ca-pbv8c" Jul 06 11:28:24.080920 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.067494 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-etc-systemd\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.080920 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.067527 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-os-release\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.080920 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.067555 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-etc-kubernetes\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.080920 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.067554 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-host-var-lib-cni-multus\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.080920 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.067599 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-etc-kubernetes\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.080920 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.067611 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-etc-systemd\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.080920 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.067633 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-var-lib-kubelet\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.080920 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.067683 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/27e80ab9-5fa0-4a48-a102-95b38f491d4f-etc-tuned\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.081532 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.067749 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8af859eb-6aa4-41b0-89cb-8028432c8f55-iptables-alerter-script\") pod \"iptables-alerter-x89qj\" (UID: \"8af859eb-6aa4-41b0-89cb-8028432c8f55\") " pod="openshift-network-operator/iptables-alerter-x89qj" Jul 06 11:28:24.081532 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.067796 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fjq9\" (UniqueName: \"kubernetes.io/projected/10439038-5c07-452f-bea0-a01a60ef437f-kube-api-access-5fjq9\") pod \"node-resolver-dw874\" (UID: \"10439038-5c07-452f-bea0-a01a60ef437f\") " pod="openshift-dns/node-resolver-dw874" Jul 06 11:28:24.081532 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.067874 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-var-lib-kubelet\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.081532 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.067918 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c3934448-4795-4cbc-8f17-8f7e14935dd2-device-dir\") pod \"aws-ebs-csi-driver-node-dh6hb\" (UID: \"c3934448-4795-4cbc-8f17-8f7e14935dd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6hb" Jul 06 11:28:24.081532 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.067959 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c3934448-4795-4cbc-8f17-8f7e14935dd2-etc-selinux\") pod \"aws-ebs-csi-driver-node-dh6hb\" (UID: \"c3934448-4795-4cbc-8f17-8f7e14935dd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6hb" Jul 06 11:28:24.081532 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.067981 2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 11:28:24.081532 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068021 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c3934448-4795-4cbc-8f17-8f7e14935dd2-sys-fs\") pod \"aws-ebs-csi-driver-node-dh6hb\" (UID: \"c3934448-4795-4cbc-8f17-8f7e14935dd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6hb" Jul 06 11:28:24.081532 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068059 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-ovnkube-script-lib\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.081532 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068069 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cfcd9856-b786-4cb9-b407-1fe4eafe748e-os-release\") pod \"multus-additional-cni-plugins-25hzg\" (UID: \"cfcd9856-b786-4cb9-b407-1fe4eafe748e\") " pod="openshift-multus/multus-additional-cni-plugins-25hzg" Jul 06 11:28:24.081532 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068138 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6kjh\" (UniqueName: \"kubernetes.io/projected/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-kube-api-access-j6kjh\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.081532 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068175 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-host-run-ovn-kubernetes\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.081532 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068257 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-ovnkube-config\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.081532 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068287 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-etc-sysctl-conf\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.081532 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068341 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-host\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.082271 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068371 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c4fa664-243d-4a56-91ce-a58f01d77768-original-pull-secret\") pod \"global-pull-secret-syncer-9mqf5\" (UID: \"9c4fa664-243d-4a56-91ce-a58f01d77768\") " pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:24.082271 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068406 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfcd9856-b786-4cb9-b407-1fe4eafe748e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-25hzg\" (UID: \"cfcd9856-b786-4cb9-b407-1fe4eafe748e\") " pod="openshift-multus/multus-additional-cni-plugins-25hzg" Jul 06 11:28:24.082271 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068426 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-host-run-ovn-kubernetes\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.082271 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068438 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-host\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.082271 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068435 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-host-run-netns\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.082271 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068485 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-host-run-netns\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.082271 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068529 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-host-cni-netd\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.082271 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068575 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-host-cni-netd\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.082271 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068607 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.082271 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068659 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-multus-cni-dir\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.082271 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068672 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-etc-sysctl-conf\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.082271 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068677 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.082271 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068709 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8af859eb-6aa4-41b0-89cb-8028432c8f55-host-slash\") pod \"iptables-alerter-x89qj\" (UID: \"8af859eb-6aa4-41b0-89cb-8028432c8f55\") " pod="openshift-network-operator/iptables-alerter-x89qj" Jul 06 11:28:24.082271 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068755 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-host-var-lib-kubelet\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.083342 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068783 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-host-slash\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.083342 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068808 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-etc-openvswitch\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.083342 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068833 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-run-openvswitch\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.083342 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068840 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-host-slash\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.083342 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068851 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-ovnkube-config\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.083342 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068857 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-node-log\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.083342 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068882 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-etc-openvswitch\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.083342 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068898 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-node-log\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.083342 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068899 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm59n\" (UniqueName: \"kubernetes.io/projected/27e80ab9-5fa0-4a48-a102-95b38f491d4f-kube-api-access-gm59n\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.083342 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068930 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-system-cni-dir\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.083342 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068955 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-host-run-netns\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.083342 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.068979 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-host-var-lib-cni-bin\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.083342 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069007 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-systemd-units\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.083342 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069040 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-run-ovn\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.083342 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069066 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e3cd243e-9aaa-4671-b9b8-2f135c474488-cni-binary-copy\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.084415 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069095 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhrdx\" (UniqueName: \"kubernetes.io/projected/c3934448-4795-4cbc-8f17-8f7e14935dd2-kube-api-access-vhrdx\") pod \"aws-ebs-csi-driver-node-dh6hb\" (UID: \"c3934448-4795-4cbc-8f17-8f7e14935dd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6hb" Jul 06 11:28:24.084415 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069124 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-run-systemd\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.084415 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069157 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-run-ovn\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.084415 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069164 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-host-cni-bin\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.084415 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069189 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27e80ab9-5fa0-4a48-a102-95b38f491d4f-tmp\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.084415 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069216 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cfcd9856-b786-4cb9-b407-1fe4eafe748e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-25hzg\" (UID: \"cfcd9856-b786-4cb9-b407-1fe4eafe748e\") " pod="openshift-multus/multus-additional-cni-plugins-25hzg" Jul 06 11:28:24.084415 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069224 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-systemd-units\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.084415 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069245 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lqxk\" (UniqueName: \"kubernetes.io/projected/cfcd9856-b786-4cb9-b407-1fe4eafe748e-kube-api-access-7lqxk\") pod \"multus-additional-cni-plugins-25hzg\" (UID: \"cfcd9856-b786-4cb9-b407-1fe4eafe748e\") " pod="openshift-multus/multus-additional-cni-plugins-25hzg" Jul 06 11:28:24.084415 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069068 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-run-openvswitch\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.084415 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069272 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-ovn-node-metrics-cert\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.084415 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069300 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-sys\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.084415 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069304 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-run-systemd\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.084415 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069319 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-host-cni-bin\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.084415 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069325 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-cnibin\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.085297 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069365 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c3934448-4795-4cbc-8f17-8f7e14935dd2-socket-dir\") pod \"aws-ebs-csi-driver-node-dh6hb\" (UID: \"c3934448-4795-4cbc-8f17-8f7e14935dd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6hb" Jul 06 11:28:24.085297 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069395 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c3934448-4795-4cbc-8f17-8f7e14935dd2-registration-dir\") pod \"aws-ebs-csi-driver-node-dh6hb\" (UID: \"c3934448-4795-4cbc-8f17-8f7e14935dd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6hb" Jul 06 11:28:24.085297 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069437 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfcd9856-b786-4cb9-b407-1fe4eafe748e-system-cni-dir\") pod \"multus-additional-cni-plugins-25hzg\" (UID: \"cfcd9856-b786-4cb9-b407-1fe4eafe748e\") " pod="openshift-multus/multus-additional-cni-plugins-25hzg" Jul 06 11:28:24.085297 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069468 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-sys\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.085297 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069465 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cfcd9856-b786-4cb9-b407-1fe4eafe748e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-25hzg\" (UID: \"cfcd9856-b786-4cb9-b407-1fe4eafe748e\") " pod="openshift-multus/multus-additional-cni-plugins-25hzg" Jul 06 11:28:24.085297 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069511 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-run\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.085297 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069535 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfxqz\" (UniqueName: \"kubernetes.io/projected/e3cd243e-9aaa-4671-b9b8-2f135c474488-kube-api-access-pfxqz\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.085297 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069561 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6730cd7d-c083-452b-b123-e2e9f92745a1-metrics-certs\") pod \"network-metrics-daemon-rgmk8\" (UID: \"6730cd7d-c083-452b-b123-e2e9f92745a1\") " pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:28:24.085297 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069586 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10439038-5c07-452f-bea0-a01a60ef437f-tmp-dir\") pod \"node-resolver-dw874\" (UID: \"10439038-5c07-452f-bea0-a01a60ef437f\") " pod="openshift-dns/node-resolver-dw874" Jul 06 11:28:24.085297 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069611 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfcd9856-b786-4cb9-b407-1fe4eafe748e-cni-binary-copy\") pod \"multus-additional-cni-plugins-25hzg\" (UID: \"cfcd9856-b786-4cb9-b407-1fe4eafe748e\") " pod="openshift-multus/multus-additional-cni-plugins-25hzg" Jul 06 11:28:24.085297 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069637 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-etc-modprobe-d\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.085297 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069668 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-var-lib-openvswitch\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.085297 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069696 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-etc-sysctl-d\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.085297 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069802 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-run\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.086110 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069818 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-etc-modprobe-d\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.086110 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069884 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-var-lib-openvswitch\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.086110 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.069937 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/27e80ab9-5fa0-4a48-a102-95b38f491d4f-etc-sysctl-d\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.086110 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.073466 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-ovn-node-metrics-cert\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.086110 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.073672 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27e80ab9-5fa0-4a48-a102-95b38f491d4f-tmp\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.086110 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.074294 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/27e80ab9-5fa0-4a48-a102-95b38f491d4f-etc-tuned\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.086110 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:24.075007 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 11:28:24.086110 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:24.075028 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 11:28:24.086110 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:24.075041 2568 projected.go:196] Error preparing data for projected volume kube-api-access-rzk6z for pod openshift-network-diagnostics/network-check-target-5dsbt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 11:28:24.086110 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:24.075118 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/685a31ed-f864-4fe8-890c-73172763af66-kube-api-access-rzk6z podName:685a31ed-f864-4fe8-890c-73172763af66 nodeName:}" failed. No retries permitted until 2026-07-06 11:28:24.575096651 +0000 UTC m=+3.054895662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rzk6z" (UniqueName: "kubernetes.io/projected/685a31ed-f864-4fe8-890c-73172763af66-kube-api-access-rzk6z") pod "network-check-target-5dsbt" (UID: "685a31ed-f864-4fe8-890c-73172763af66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 11:28:24.086110 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.077305 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm59n\" (UniqueName: \"kubernetes.io/projected/27e80ab9-5fa0-4a48-a102-95b38f491d4f-kube-api-access-gm59n\") pod \"tuned-54f75\" (UID: \"27e80ab9-5fa0-4a48-a102-95b38f491d4f\") " pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.086110 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.077739 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhrxq\" (UniqueName: \"kubernetes.io/projected/733213c7-49ab-4f45-a0dd-d87abe31ef28-kube-api-access-xhrxq\") pod \"node-ca-pbv8c\" (UID: \"733213c7-49ab-4f45-a0dd-d87abe31ef28\") " pod="openshift-image-registry/node-ca-pbv8c" Jul 06 11:28:24.086110 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.077833 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6kjh\" (UniqueName: \"kubernetes.io/projected/6c7bab31-dddb-4e05-a2c6-5babd5e779d4-kube-api-access-j6kjh\") pod \"ovnkube-node-tn42c\" (UID: \"6c7bab31-dddb-4e05-a2c6-5babd5e779d4\") " pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.177392 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.170840 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-host-var-lib-kubelet\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.177392 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.170892 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-system-cni-dir\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.177392 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.170920 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-host-run-netns\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.177392 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.170948 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-host-var-lib-cni-bin\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.177392 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.170984 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e3cd243e-9aaa-4671-b9b8-2f135c474488-cni-binary-copy\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.177392 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.171019 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhrdx\" (UniqueName: \"kubernetes.io/projected/c3934448-4795-4cbc-8f17-8f7e14935dd2-kube-api-access-vhrdx\") pod \"aws-ebs-csi-driver-node-dh6hb\" (UID: \"c3934448-4795-4cbc-8f17-8f7e14935dd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6hb" Jul 06 11:28:24.177392 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.171048 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cfcd9856-b786-4cb9-b407-1fe4eafe748e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-25hzg\" (UID: \"cfcd9856-b786-4cb9-b407-1fe4eafe748e\") " pod="openshift-multus/multus-additional-cni-plugins-25hzg" Jul 06 11:28:24.177392 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.171067 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-host-run-netns\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.177392 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.171079 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lqxk\" (UniqueName: \"kubernetes.io/projected/cfcd9856-b786-4cb9-b407-1fe4eafe748e-kube-api-access-7lqxk\") pod \"multus-additional-cni-plugins-25hzg\" (UID: \"cfcd9856-b786-4cb9-b407-1fe4eafe748e\") " pod="openshift-multus/multus-additional-cni-plugins-25hzg" Jul 06 11:28:24.177392 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.171075 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-system-cni-dir\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.177392 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.171109 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-cnibin\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.177392 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.171133 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c3934448-4795-4cbc-8f17-8f7e14935dd2-socket-dir\") pod \"aws-ebs-csi-driver-node-dh6hb\" (UID: \"c3934448-4795-4cbc-8f17-8f7e14935dd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6hb" Jul 06 11:28:24.177392 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.171156 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c3934448-4795-4cbc-8f17-8f7e14935dd2-registration-dir\") pod \"aws-ebs-csi-driver-node-dh6hb\" (UID: \"c3934448-4795-4cbc-8f17-8f7e14935dd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6hb" Jul 06 11:28:24.177392 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.171180 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfcd9856-b786-4cb9-b407-1fe4eafe748e-system-cni-dir\") pod \"multus-additional-cni-plugins-25hzg\" (UID: \"cfcd9856-b786-4cb9-b407-1fe4eafe748e\") " pod="openshift-multus/multus-additional-cni-plugins-25hzg" Jul 06 11:28:24.179185 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.171206 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cfcd9856-b786-4cb9-b407-1fe4eafe748e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-25hzg\" (UID: \"cfcd9856-b786-4cb9-b407-1fe4eafe748e\") " pod="openshift-multus/multus-additional-cni-plugins-25hzg" Jul 06 11:28:24.179185 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.171233 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfxqz\" (UniqueName: \"kubernetes.io/projected/e3cd243e-9aaa-4671-b9b8-2f135c474488-kube-api-access-pfxqz\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.179185 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.171390 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c3934448-4795-4cbc-8f17-8f7e14935dd2-registration-dir\") pod \"aws-ebs-csi-driver-node-dh6hb\" (UID: \"c3934448-4795-4cbc-8f17-8f7e14935dd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6hb" Jul 06 11:28:24.179185 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.171670 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-cnibin\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.179185 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.171813 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c3934448-4795-4cbc-8f17-8f7e14935dd2-socket-dir\") pod \"aws-ebs-csi-driver-node-dh6hb\" (UID: \"c3934448-4795-4cbc-8f17-8f7e14935dd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6hb" Jul 06 11:28:24.179185 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.171887 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cfcd9856-b786-4cb9-b407-1fe4eafe748e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-25hzg\" (UID: \"cfcd9856-b786-4cb9-b407-1fe4eafe748e\") " pod="openshift-multus/multus-additional-cni-plugins-25hzg" Jul 06 11:28:24.179185 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.172155 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfcd9856-b786-4cb9-b407-1fe4eafe748e-system-cni-dir\") pod \"multus-additional-cni-plugins-25hzg\" (UID: \"cfcd9856-b786-4cb9-b407-1fe4eafe748e\") " pod="openshift-multus/multus-additional-cni-plugins-25hzg" Jul 06 11:28:24.179185 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.172286 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cfcd9856-b786-4cb9-b407-1fe4eafe748e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-25hzg\" (UID: \"cfcd9856-b786-4cb9-b407-1fe4eafe748e\") " pod="openshift-multus/multus-additional-cni-plugins-25hzg" Jul 06 11:28:24.179185 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:24.172334 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 11:28:24.179185 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:24.172400 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6730cd7d-c083-452b-b123-e2e9f92745a1-metrics-certs podName:6730cd7d-c083-452b-b123-e2e9f92745a1 nodeName:}" failed. No retries permitted until 2026-07-06 11:28:24.672382154 +0000 UTC m=+3.152181172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6730cd7d-c083-452b-b123-e2e9f92745a1-metrics-certs") pod "network-metrics-daemon-rgmk8" (UID: "6730cd7d-c083-452b-b123-e2e9f92745a1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 11:28:24.179185 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.173725 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e3cd243e-9aaa-4671-b9b8-2f135c474488-cni-binary-copy\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.188318 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.171711 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-host-var-lib-cni-bin\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.188318 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.186522 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6730cd7d-c083-452b-b123-e2e9f92745a1-metrics-certs\") pod \"network-metrics-daemon-rgmk8\" (UID: \"6730cd7d-c083-452b-b123-e2e9f92745a1\") " pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:28:24.188318 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.186577 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10439038-5c07-452f-bea0-a01a60ef437f-tmp-dir\") pod \"node-resolver-dw874\" (UID: \"10439038-5c07-452f-bea0-a01a60ef437f\") " pod="openshift-dns/node-resolver-dw874" Jul 06 11:28:24.188318 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.186609 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfcd9856-b786-4cb9-b407-1fe4eafe748e-cni-binary-copy\") pod \"multus-additional-cni-plugins-25hzg\" (UID: \"cfcd9856-b786-4cb9-b407-1fe4eafe748e\") " pod="openshift-multus/multus-additional-cni-plugins-25hzg" Jul 06 11:28:24.188318 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.186952 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10439038-5c07-452f-bea0-a01a60ef437f-tmp-dir\") pod \"node-resolver-dw874\" (UID: \"10439038-5c07-452f-bea0-a01a60ef437f\") " pod="openshift-dns/node-resolver-dw874" Jul 06 11:28:24.188318 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.187860 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfcd9856-b786-4cb9-b407-1fe4eafe748e-cni-binary-copy\") pod \"multus-additional-cni-plugins-25hzg\" (UID: \"cfcd9856-b786-4cb9-b407-1fe4eafe748e\") " pod="openshift-multus/multus-additional-cni-plugins-25hzg" Jul 06 11:28:24.188318 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.187983 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-host-var-lib-kubelet\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.188318 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.188039 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-host-run-multus-certs\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.188318 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.188115 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c4fa664-243d-4a56-91ce-a58f01d77768-kubelet-config\") pod \"global-pull-secret-syncer-9mqf5\" (UID: \"9c4fa664-243d-4a56-91ce-a58f01d77768\") " pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:24.188318 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.188193 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-host-run-multus-certs\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.188318 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.188223 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c4fa664-243d-4a56-91ce-a58f01d77768-dbus\") pod \"global-pull-secret-syncer-9mqf5\" (UID: \"9c4fa664-243d-4a56-91ce-a58f01d77768\") " pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:24.188318 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.188226 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c4fa664-243d-4a56-91ce-a58f01d77768-kubelet-config\") pod \"global-pull-secret-syncer-9mqf5\" (UID: \"9c4fa664-243d-4a56-91ce-a58f01d77768\") " pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:24.188318 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.188269 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/10439038-5c07-452f-bea0-a01a60ef437f-hosts-file\") pod \"node-resolver-dw874\" (UID: \"10439038-5c07-452f-bea0-a01a60ef437f\") " pod="openshift-dns/node-resolver-dw874" Jul 06 11:28:24.188318 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.188305 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-hostroot\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.189275 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.188359 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-multus-conf-dir\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.189275 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.188380 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/10439038-5c07-452f-bea0-a01a60ef437f-hosts-file\") pod \"node-resolver-dw874\" (UID: \"10439038-5c07-452f-bea0-a01a60ef437f\") " pod="openshift-dns/node-resolver-dw874" Jul 06 11:28:24.189275 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.188392 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kplxs\" (UniqueName: \"kubernetes.io/projected/6730cd7d-c083-452b-b123-e2e9f92745a1-kube-api-access-kplxs\") pod \"network-metrics-daemon-rgmk8\" (UID: \"6730cd7d-c083-452b-b123-e2e9f92745a1\") " pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:28:24.189275 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.188308 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c4fa664-243d-4a56-91ce-a58f01d77768-dbus\") pod \"global-pull-secret-syncer-9mqf5\" (UID: \"9c4fa664-243d-4a56-91ce-a58f01d77768\") " pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:24.189275 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.188439 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/218f64a8-8fd8-462b-972d-ff9ecedb6a74-konnectivity-ca\") pod \"konnectivity-agent-l89bb\" (UID: \"218f64a8-8fd8-462b-972d-ff9ecedb6a74\") " pod="kube-system/konnectivity-agent-l89bb" Jul 06 11:28:24.189275 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.188467 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-multus-conf-dir\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.189275 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.188442 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-hostroot\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.189275 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.188523 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c3934448-4795-4cbc-8f17-8f7e14935dd2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dh6hb\" (UID: \"c3934448-4795-4cbc-8f17-8f7e14935dd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6hb" Jul 06 11:28:24.189275 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.188471 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c3934448-4795-4cbc-8f17-8f7e14935dd2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dh6hb\" (UID: \"c3934448-4795-4cbc-8f17-8f7e14935dd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6hb" Jul 06 11:28:24.189275 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.188563 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e3cd243e-9aaa-4671-b9b8-2f135c474488-multus-daemon-config\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.189275 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.188592 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4vnz\" (UniqueName: \"kubernetes.io/projected/8af859eb-6aa4-41b0-89cb-8028432c8f55-kube-api-access-z4vnz\") pod \"iptables-alerter-x89qj\" (UID: \"8af859eb-6aa4-41b0-89cb-8028432c8f55\") " pod="openshift-network-operator/iptables-alerter-x89qj" Jul 06 11:28:24.189275 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.188618 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-host-run-k8s-cni-cncf-io\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.189275 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.188665 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfcd9856-b786-4cb9-b407-1fe4eafe748e-cnibin\") pod \"multus-additional-cni-plugins-25hzg\" (UID: \"cfcd9856-b786-4cb9-b407-1fe4eafe748e\") " pod="openshift-multus/multus-additional-cni-plugins-25hzg" Jul 06 11:28:24.189275 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.188692 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-multus-socket-dir-parent\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.189275 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.188741 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/218f64a8-8fd8-462b-972d-ff9ecedb6a74-agent-certs\") pod \"konnectivity-agent-l89bb\" (UID: \"218f64a8-8fd8-462b-972d-ff9ecedb6a74\") " pod="kube-system/konnectivity-agent-l89bb" Jul 06 11:28:24.191049 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.188772 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-os-release\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.191049 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.188784 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-host-run-k8s-cni-cncf-io\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.191049 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.188798 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-host-var-lib-cni-multus\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.191049 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.188826 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-etc-kubernetes\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.191049 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.188859 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8af859eb-6aa4-41b0-89cb-8028432c8f55-iptables-alerter-script\") pod \"iptables-alerter-x89qj\" (UID: \"8af859eb-6aa4-41b0-89cb-8028432c8f55\") " pod="openshift-network-operator/iptables-alerter-x89qj" Jul 06 11:28:24.191049 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.189105 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-os-release\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.191049 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.189156 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfcd9856-b786-4cb9-b407-1fe4eafe748e-cnibin\") pod \"multus-additional-cni-plugins-25hzg\" (UID: \"cfcd9856-b786-4cb9-b407-1fe4eafe748e\") " pod="openshift-multus/multus-additional-cni-plugins-25hzg" Jul 06 11:28:24.191049 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.189189 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/218f64a8-8fd8-462b-972d-ff9ecedb6a74-konnectivity-ca\") pod \"konnectivity-agent-l89bb\" (UID: \"218f64a8-8fd8-462b-972d-ff9ecedb6a74\") " pod="kube-system/konnectivity-agent-l89bb" Jul 06 11:28:24.191049 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.189218 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-multus-socket-dir-parent\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.191049 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.189407 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fjq9\" (UniqueName: \"kubernetes.io/projected/10439038-5c07-452f-bea0-a01a60ef437f-kube-api-access-5fjq9\") pod \"node-resolver-dw874\" (UID: \"10439038-5c07-452f-bea0-a01a60ef437f\") " pod="openshift-dns/node-resolver-dw874" Jul 06 11:28:24.191049 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.189435 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c3934448-4795-4cbc-8f17-8f7e14935dd2-device-dir\") pod \"aws-ebs-csi-driver-node-dh6hb\" (UID: \"c3934448-4795-4cbc-8f17-8f7e14935dd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6hb" Jul 06 11:28:24.191049 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.189459 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c3934448-4795-4cbc-8f17-8f7e14935dd2-etc-selinux\") pod \"aws-ebs-csi-driver-node-dh6hb\" (UID: \"c3934448-4795-4cbc-8f17-8f7e14935dd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6hb" Jul 06 11:28:24.191049 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.189482 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c3934448-4795-4cbc-8f17-8f7e14935dd2-sys-fs\") pod \"aws-ebs-csi-driver-node-dh6hb\" (UID: \"c3934448-4795-4cbc-8f17-8f7e14935dd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6hb" Jul 06 11:28:24.191049 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.189511 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cfcd9856-b786-4cb9-b407-1fe4eafe748e-os-release\") pod \"multus-additional-cni-plugins-25hzg\" (UID: \"cfcd9856-b786-4cb9-b407-1fe4eafe748e\") " pod="openshift-multus/multus-additional-cni-plugins-25hzg" Jul 06 11:28:24.192281 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.189551 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c4fa664-243d-4a56-91ce-a58f01d77768-original-pull-secret\") pod \"global-pull-secret-syncer-9mqf5\" (UID: \"9c4fa664-243d-4a56-91ce-a58f01d77768\") " pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:24.192281 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.189579 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfcd9856-b786-4cb9-b407-1fe4eafe748e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-25hzg\" (UID: \"cfcd9856-b786-4cb9-b407-1fe4eafe748e\") " pod="openshift-multus/multus-additional-cni-plugins-25hzg" Jul 06 11:28:24.192281 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.189608 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-multus-cni-dir\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.192281 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.189634 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8af859eb-6aa4-41b0-89cb-8028432c8f55-host-slash\") pod \"iptables-alerter-x89qj\" (UID: \"8af859eb-6aa4-41b0-89cb-8028432c8f55\") " pod="openshift-network-operator/iptables-alerter-x89qj" Jul 06 11:28:24.192281 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.189710 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8af859eb-6aa4-41b0-89cb-8028432c8f55-host-slash\") pod \"iptables-alerter-x89qj\" (UID: \"8af859eb-6aa4-41b0-89cb-8028432c8f55\") " pod="openshift-network-operator/iptables-alerter-x89qj" Jul 06 11:28:24.192281 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.189802 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e3cd243e-9aaa-4671-b9b8-2f135c474488-multus-daemon-config\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.192281 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.189867 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-host-var-lib-cni-multus\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.192281 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.189924 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-etc-kubernetes\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.192281 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.189980 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c3934448-4795-4cbc-8f17-8f7e14935dd2-sys-fs\") pod \"aws-ebs-csi-driver-node-dh6hb\" (UID: \"c3934448-4795-4cbc-8f17-8f7e14935dd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6hb" Jul 06 11:28:24.192281 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.190007 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cfcd9856-b786-4cb9-b407-1fe4eafe748e-os-release\") pod \"multus-additional-cni-plugins-25hzg\" (UID: \"cfcd9856-b786-4cb9-b407-1fe4eafe748e\") " pod="openshift-multus/multus-additional-cni-plugins-25hzg" Jul 06 11:28:24.192281 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.190034 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c3934448-4795-4cbc-8f17-8f7e14935dd2-device-dir\") pod \"aws-ebs-csi-driver-node-dh6hb\" (UID: \"c3934448-4795-4cbc-8f17-8f7e14935dd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6hb" Jul 06 11:28:24.192281 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:24.190090 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 11:28:24.192281 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.190142 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c3934448-4795-4cbc-8f17-8f7e14935dd2-etc-selinux\") pod \"aws-ebs-csi-driver-node-dh6hb\" (UID: \"c3934448-4795-4cbc-8f17-8f7e14935dd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6hb" Jul 06 11:28:24.192281 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:24.190148 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c4fa664-243d-4a56-91ce-a58f01d77768-original-pull-secret podName:9c4fa664-243d-4a56-91ce-a58f01d77768 nodeName:}" failed. No retries permitted until 2026-07-06 11:28:24.69012968 +0000 UTC m=+3.169928676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c4fa664-243d-4a56-91ce-a58f01d77768-original-pull-secret") pod "global-pull-secret-syncer-9mqf5" (UID: "9c4fa664-243d-4a56-91ce-a58f01d77768") : object "kube-system"/"original-pull-secret" not registered Jul 06 11:28:24.314168 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.190213 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3cd243e-9aaa-4671-b9b8-2f135c474488-multus-cni-dir\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.314168 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.190297 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfcd9856-b786-4cb9-b407-1fe4eafe748e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-25hzg\" (UID: \"cfcd9856-b786-4cb9-b407-1fe4eafe748e\") " pod="openshift-multus/multus-additional-cni-plugins-25hzg" Jul 06 11:28:24.314168 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.190384 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8af859eb-6aa4-41b0-89cb-8028432c8f55-iptables-alerter-script\") pod \"iptables-alerter-x89qj\" (UID: \"8af859eb-6aa4-41b0-89cb-8028432c8f55\") " pod="openshift-network-operator/iptables-alerter-x89qj" Jul 06 11:28:24.314168 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.193027 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/218f64a8-8fd8-462b-972d-ff9ecedb6a74-agent-certs\") pod \"konnectivity-agent-l89bb\" (UID: \"218f64a8-8fd8-462b-972d-ff9ecedb6a74\") " pod="kube-system/konnectivity-agent-l89bb" Jul 06 11:28:24.314168 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.193809 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhrdx\" (UniqueName: \"kubernetes.io/projected/c3934448-4795-4cbc-8f17-8f7e14935dd2-kube-api-access-vhrdx\") pod \"aws-ebs-csi-driver-node-dh6hb\" (UID: \"c3934448-4795-4cbc-8f17-8f7e14935dd2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6hb" Jul 06 11:28:24.314168 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.196898 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfxqz\" (UniqueName: \"kubernetes.io/projected/e3cd243e-9aaa-4671-b9b8-2f135c474488-kube-api-access-pfxqz\") pod \"multus-pvz5d\" (UID: \"e3cd243e-9aaa-4671-b9b8-2f135c474488\") " pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.314168 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.198430 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lqxk\" (UniqueName: \"kubernetes.io/projected/cfcd9856-b786-4cb9-b407-1fe4eafe748e-kube-api-access-7lqxk\") pod \"multus-additional-cni-plugins-25hzg\" (UID: \"cfcd9856-b786-4cb9-b407-1fe4eafe748e\") " pod="openshift-multus/multus-additional-cni-plugins-25hzg" Jul 06 11:28:24.314168 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.201991 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kplxs\" (UniqueName: \"kubernetes.io/projected/6730cd7d-c083-452b-b123-e2e9f92745a1-kube-api-access-kplxs\") pod \"network-metrics-daemon-rgmk8\" (UID: \"6730cd7d-c083-452b-b123-e2e9f92745a1\") " pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:28:24.314168 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.204143 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fjq9\" (UniqueName: \"kubernetes.io/projected/10439038-5c07-452f-bea0-a01a60ef437f-kube-api-access-5fjq9\") pod \"node-resolver-dw874\" (UID: \"10439038-5c07-452f-bea0-a01a60ef437f\") " pod="openshift-dns/node-resolver-dw874" Jul 06 11:28:24.314168 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.208341 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4vnz\" (UniqueName: \"kubernetes.io/projected/8af859eb-6aa4-41b0-89cb-8028432c8f55-kube-api-access-z4vnz\") pod \"iptables-alerter-x89qj\" (UID: \"8af859eb-6aa4-41b0-89cb-8028432c8f55\") " pod="openshift-network-operator/iptables-alerter-x89qj" Jul 06 11:28:24.314168 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.249841 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-54f75" Jul 06 11:28:24.314168 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.262117 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pbv8c" Jul 06 11:28:24.314168 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:24.263608 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27e80ab9_5fa0_4a48_a102_95b38f491d4f.slice/crio-f39b615e28791f0d657750d255051a7a168d8cd3fbbdd97c905da2662aa9b4c7 WatchSource:0}: Error finding container f39b615e28791f0d657750d255051a7a168d8cd3fbbdd97c905da2662aa9b4c7: Status 404 returned error can't find the container with id f39b615e28791f0d657750d255051a7a168d8cd3fbbdd97c905da2662aa9b4c7 Jul 06 11:28:24.314168 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:24.272801 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod733213c7_49ab_4f45_a0dd_d87abe31ef28.slice/crio-062acbdf814ccc9f7214d2c041d2f1341f6e106008b51375d8edf3b84a795f63 WatchSource:0}: Error finding container 062acbdf814ccc9f7214d2c041d2f1341f6e106008b51375d8edf3b84a795f63: Status 404 returned error can't find the container with id 062acbdf814ccc9f7214d2c041d2f1341f6e106008b51375d8edf3b84a795f63 Jul 06 11:28:24.314168 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.273691 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:24.314168 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.278122 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pvz5d" Jul 06 11:28:24.317041 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:24.284613 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c7bab31_dddb_4e05_a2c6_5babd5e779d4.slice/crio-6e1a2fac32c4e5f39e878e664d08876b9ecb4c50cce35973bff30d3aee082835 WatchSource:0}: Error finding container 6e1a2fac32c4e5f39e878e664d08876b9ecb4c50cce35973bff30d3aee082835: Status 404 returned error can't find the container with id 6e1a2fac32c4e5f39e878e664d08876b9ecb4c50cce35973bff30d3aee082835 Jul 06 11:28:24.317041 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.288889 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6hb" Jul 06 11:28:24.317041 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:24.289391 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3cd243e_9aaa_4671_b9b8_2f135c474488.slice/crio-dc779c66b6b627aa53f42b02fe97c46b196b14ac4eaf3a2ecd70f302a047fe7a WatchSource:0}: Error finding container dc779c66b6b627aa53f42b02fe97c46b196b14ac4eaf3a2ecd70f302a047fe7a: Status 404 returned error can't find the container with id dc779c66b6b627aa53f42b02fe97c46b196b14ac4eaf3a2ecd70f302a047fe7a Jul 06 11:28:24.317041 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.302915 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x89qj" Jul 06 11:28:24.317041 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:24.306588 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3934448_4795_4cbc_8f17_8f7e14935dd2.slice/crio-c02c8d17d2a2fbb62344402b894a1ba2bba1d723f416a2f8d5e693c8ca0311c2 WatchSource:0}: Error finding container c02c8d17d2a2fbb62344402b894a1ba2bba1d723f416a2f8d5e693c8ca0311c2: Status 404 returned error can't find the container with id c02c8d17d2a2fbb62344402b894a1ba2bba1d723f416a2f8d5e693c8ca0311c2 Jul 06 11:28:24.317041 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:24.313683 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8af859eb_6aa4_41b0_89cb_8028432c8f55.slice/crio-0b701ae3374e2eb9064c189e51919e4f47d24557d22e87c0b26207d182ef720b WatchSource:0}: Error finding container 0b701ae3374e2eb9064c189e51919e4f47d24557d22e87c0b26207d182ef720b: Status 404 returned error can't find the container with id 0b701ae3374e2eb9064c189e51919e4f47d24557d22e87c0b26207d182ef720b Jul 06 11:28:24.317041 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.315093 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l89bb" Jul 06 11:28:24.470307 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.324672 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dw874" Jul 06 11:28:24.470307 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.334486 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-25hzg" Jul 06 11:28:24.470307 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.418912 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 11:28:24.622766 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.592536 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzk6z\" (UniqueName: \"kubernetes.io/projected/685a31ed-f864-4fe8-890c-73172763af66-kube-api-access-rzk6z\") pod \"network-check-target-5dsbt\" (UID: \"685a31ed-f864-4fe8-890c-73172763af66\") " pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:28:24.622766 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:24.592732 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 11:28:24.622766 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:24.592753 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 11:28:24.622766 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:24.592764 2568 projected.go:196] Error preparing data for projected volume kube-api-access-rzk6z for pod openshift-network-diagnostics/network-check-target-5dsbt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 11:28:24.622766 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:24.592823 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/685a31ed-f864-4fe8-890c-73172763af66-kube-api-access-rzk6z podName:685a31ed-f864-4fe8-890c-73172763af66 nodeName:}" failed. No retries permitted until 2026-07-06 11:28:25.592808251 +0000 UTC m=+4.072607247 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rzk6z" (UniqueName: "kubernetes.io/projected/685a31ed-f864-4fe8-890c-73172763af66-kube-api-access-rzk6z") pod "network-check-target-5dsbt" (UID: "685a31ed-f864-4fe8-890c-73172763af66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 11:28:24.813054 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.693441 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6730cd7d-c083-452b-b123-e2e9f92745a1-metrics-certs\") pod \"network-metrics-daemon-rgmk8\" (UID: \"6730cd7d-c083-452b-b123-e2e9f92745a1\") " pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:28:24.813054 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.693524 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c4fa664-243d-4a56-91ce-a58f01d77768-original-pull-secret\") pod \"global-pull-secret-syncer-9mqf5\" (UID: \"9c4fa664-243d-4a56-91ce-a58f01d77768\") " pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:24.813054 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:24.693607 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 11:28:24.813054 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:24.693617 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 11:28:24.813054 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:24.693678 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6730cd7d-c083-452b-b123-e2e9f92745a1-metrics-certs podName:6730cd7d-c083-452b-b123-e2e9f92745a1 nodeName:}" failed. No retries permitted until 2026-07-06 11:28:25.693663534 +0000 UTC m=+4.173462533 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6730cd7d-c083-452b-b123-e2e9f92745a1-metrics-certs") pod "network-metrics-daemon-rgmk8" (UID: "6730cd7d-c083-452b-b123-e2e9f92745a1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 11:28:24.813054 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:24.693711 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c4fa664-243d-4a56-91ce-a58f01d77768-original-pull-secret podName:9c4fa664-243d-4a56-91ce-a58f01d77768 nodeName:}" failed. No retries permitted until 2026-07-06 11:28:25.693687954 +0000 UTC m=+4.173486949 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c4fa664-243d-4a56-91ce-a58f01d77768-original-pull-secret") pod "global-pull-secret-syncer-9mqf5" (UID: "9c4fa664-243d-4a56-91ce-a58f01d77768") : object "kube-system"/"original-pull-secret" not registered Jul 06 11:28:24.813054 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:24.739940 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod218f64a8_8fd8_462b_972d_ff9ecedb6a74.slice/crio-3aaa73ed3d386c2b0385e97f1cbe8010e09c49d04d4679120a1dec4b49ef78d2 WatchSource:0}: Error finding container 3aaa73ed3d386c2b0385e97f1cbe8010e09c49d04d4679120a1dec4b49ef78d2: Status 404 returned error can't find the container with id 3aaa73ed3d386c2b0385e97f1cbe8010e09c49d04d4679120a1dec4b49ef78d2 Jul 06 11:28:24.989553 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.989476 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 11:23:24 +0000 UTC" deadline="2028-02-21 19:59:28.902409369 +0000 UTC" Jul 06 11:28:24.989553 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:24.989552 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14288h31m3.912863352s" Jul 06 11:28:25.046528 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:25.046487 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:28:25.046754 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:25.046650 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dsbt" podUID="685a31ed-f864-4fe8-890c-73172763af66" Jul 06 11:28:25.056628 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:25.056590 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-54f75" event={"ID":"27e80ab9-5fa0-4a48-a102-95b38f491d4f","Type":"ContainerStarted","Data":"f39b615e28791f0d657750d255051a7a168d8cd3fbbdd97c905da2662aa9b4c7"} Jul 06 11:28:25.058111 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:25.058041 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l89bb" event={"ID":"218f64a8-8fd8-462b-972d-ff9ecedb6a74","Type":"ContainerStarted","Data":"3aaa73ed3d386c2b0385e97f1cbe8010e09c49d04d4679120a1dec4b49ef78d2"} Jul 06 11:28:25.059417 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:25.059388 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6hb" event={"ID":"c3934448-4795-4cbc-8f17-8f7e14935dd2","Type":"ContainerStarted","Data":"c02c8d17d2a2fbb62344402b894a1ba2bba1d723f416a2f8d5e693c8ca0311c2"} Jul 06 11:28:25.060862 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:25.060838 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pvz5d" event={"ID":"e3cd243e-9aaa-4671-b9b8-2f135c474488","Type":"ContainerStarted","Data":"dc779c66b6b627aa53f42b02fe97c46b196b14ac4eaf3a2ecd70f302a047fe7a"} Jul 06 11:28:25.062522 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:25.062496 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" event={"ID":"6c7bab31-dddb-4e05-a2c6-5babd5e779d4","Type":"ContainerStarted","Data":"6e1a2fac32c4e5f39e878e664d08876b9ecb4c50cce35973bff30d3aee082835"} Jul 06 11:28:25.063994 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:25.063929 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x89qj" event={"ID":"8af859eb-6aa4-41b0-89cb-8028432c8f55","Type":"ContainerStarted","Data":"0b701ae3374e2eb9064c189e51919e4f47d24557d22e87c0b26207d182ef720b"} Jul 06 11:28:25.065223 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:25.065196 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pbv8c" event={"ID":"733213c7-49ab-4f45-a0dd-d87abe31ef28","Type":"ContainerStarted","Data":"062acbdf814ccc9f7214d2c041d2f1341f6e106008b51375d8edf3b84a795f63"} Jul 06 11:28:25.601231 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:25.601189 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzk6z\" (UniqueName: \"kubernetes.io/projected/685a31ed-f864-4fe8-890c-73172763af66-kube-api-access-rzk6z\") pod \"network-check-target-5dsbt\" (UID: \"685a31ed-f864-4fe8-890c-73172763af66\") " pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:28:25.602060 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:25.601383 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 11:28:25.602060 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:25.601405 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 11:28:25.602060 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:25.601421 2568 projected.go:196] Error preparing data for projected volume kube-api-access-rzk6z for pod openshift-network-diagnostics/network-check-target-5dsbt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 11:28:25.602060 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:25.601489 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/685a31ed-f864-4fe8-890c-73172763af66-kube-api-access-rzk6z podName:685a31ed-f864-4fe8-890c-73172763af66 nodeName:}" failed. No retries permitted until 2026-07-06 11:28:27.601467843 +0000 UTC m=+6.081266912 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rzk6z" (UniqueName: "kubernetes.io/projected/685a31ed-f864-4fe8-890c-73172763af66-kube-api-access-rzk6z") pod "network-check-target-5dsbt" (UID: "685a31ed-f864-4fe8-890c-73172763af66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 11:28:25.701611 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:25.701567 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c4fa664-243d-4a56-91ce-a58f01d77768-original-pull-secret\") pod \"global-pull-secret-syncer-9mqf5\" (UID: \"9c4fa664-243d-4a56-91ce-a58f01d77768\") " pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:25.701838 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:25.701689 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6730cd7d-c083-452b-b123-e2e9f92745a1-metrics-certs\") pod \"network-metrics-daemon-rgmk8\" (UID: \"6730cd7d-c083-452b-b123-e2e9f92745a1\") " pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:28:25.701838 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:25.701779 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 11:28:25.701956 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:25.701864 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c4fa664-243d-4a56-91ce-a58f01d77768-original-pull-secret podName:9c4fa664-243d-4a56-91ce-a58f01d77768 nodeName:}" failed. No retries permitted until 2026-07-06 11:28:27.701841458 +0000 UTC m=+6.181640470 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c4fa664-243d-4a56-91ce-a58f01d77768-original-pull-secret") pod "global-pull-secret-syncer-9mqf5" (UID: "9c4fa664-243d-4a56-91ce-a58f01d77768") : object "kube-system"/"original-pull-secret" not registered Jul 06 11:28:25.701956 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:25.701875 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 11:28:25.701956 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:25.701936 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6730cd7d-c083-452b-b123-e2e9f92745a1-metrics-certs podName:6730cd7d-c083-452b-b123-e2e9f92745a1 nodeName:}" failed. No retries permitted until 2026-07-06 11:28:27.701918902 +0000 UTC m=+6.181717913 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6730cd7d-c083-452b-b123-e2e9f92745a1-metrics-certs") pod "network-metrics-daemon-rgmk8" (UID: "6730cd7d-c083-452b-b123-e2e9f92745a1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 11:28:25.855099 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:25.854928 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10439038_5c07_452f_bea0_a01a60ef437f.slice/crio-bf2bf21fc7092b5c2943818e2d11da9aa17e6625637f181ca705764edeb2e423 WatchSource:0}: Error finding container bf2bf21fc7092b5c2943818e2d11da9aa17e6625637f181ca705764edeb2e423: Status 404 returned error can't find the container with id bf2bf21fc7092b5c2943818e2d11da9aa17e6625637f181ca705764edeb2e423 Jul 06 11:28:25.864096 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:25.864040 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfcd9856_b786_4cb9_b407_1fe4eafe748e.slice/crio-5cd8ccd6e19107774a6c1591cbdde5d92d4ecdb4666b71bba920a1f2a7d1dc18 WatchSource:0}: Error finding container 5cd8ccd6e19107774a6c1591cbdde5d92d4ecdb4666b71bba920a1f2a7d1dc18: Status 404 returned error can't find the container with id 5cd8ccd6e19107774a6c1591cbdde5d92d4ecdb4666b71bba920a1f2a7d1dc18 Jul 06 11:28:25.989911 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:25.989834 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 11:23:24 +0000 UTC" deadline="2028-01-22 09:05:58.70379667 +0000 UTC" Jul 06 11:28:25.989911 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:25.989878 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13557h37m32.713924468s" Jul 06 11:28:26.049688 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:26.049192 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:26.049688 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:26.049336 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9mqf5" podUID="9c4fa664-243d-4a56-91ce-a58f01d77768" Jul 06 11:28:26.049688 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:26.049522 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:28:26.049688 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:26.049624 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgmk8" podUID="6730cd7d-c083-452b-b123-e2e9f92745a1" Jul 06 11:28:26.074563 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:26.074503 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dw874" event={"ID":"10439038-5c07-452f-bea0-a01a60ef437f","Type":"ContainerStarted","Data":"bf2bf21fc7092b5c2943818e2d11da9aa17e6625637f181ca705764edeb2e423"} Jul 06 11:28:26.078640 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:26.078553 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-25hzg" event={"ID":"cfcd9856-b786-4cb9-b407-1fe4eafe748e","Type":"ContainerStarted","Data":"5cd8ccd6e19107774a6c1591cbdde5d92d4ecdb4666b71bba920a1f2a7d1dc18"} Jul 06 11:28:26.085020 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:26.084933 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-176.ec2.internal" event={"ID":"cf4e10fd22892d39b2d489b025b34f3d","Type":"ContainerStarted","Data":"73355b4a90aef0c41b51233f0e5ce82bb1980bb3a30b0eedceb374396219f4ca"} Jul 06 11:28:26.103400 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:26.103308 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-176.ec2.internal" podStartSLOduration=2.103292697 podStartE2EDuration="2.103292697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 11:28:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:28:26.102831031 +0000 UTC m=+4.582630045" watchObservedRunningTime="2026-07-06 11:28:26.103292697 +0000 UTC m=+4.583091716" Jul 06 11:28:27.046246 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:27.046145 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:28:27.046745 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:27.046305 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dsbt" podUID="685a31ed-f864-4fe8-890c-73172763af66" Jul 06 11:28:27.096743 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:27.096703 2568 generic.go:356] "Generic (PLEG): container finished" podID="bb7589110c767d3053aba8654c931fec" containerID="13b9162a2e5c4c9b8a7b34730fa1b02919fce620f582cecada0d7561d04fcaef" exitCode=0 Jul 06 11:28:27.096946 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:27.096915 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-176.ec2.internal" event={"ID":"bb7589110c767d3053aba8654c931fec","Type":"ContainerDied","Data":"13b9162a2e5c4c9b8a7b34730fa1b02919fce620f582cecada0d7561d04fcaef"} Jul 06 11:28:27.622120 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:27.622025 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzk6z\" (UniqueName: \"kubernetes.io/projected/685a31ed-f864-4fe8-890c-73172763af66-kube-api-access-rzk6z\") pod \"network-check-target-5dsbt\" (UID: \"685a31ed-f864-4fe8-890c-73172763af66\") " pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:28:27.622311 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:27.622239 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 11:28:27.622311 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:27.622271 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 11:28:27.622311 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:27.622289 2568 projected.go:196] Error preparing data for projected volume kube-api-access-rzk6z for pod openshift-network-diagnostics/network-check-target-5dsbt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 11:28:27.622488 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:27.622371 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/685a31ed-f864-4fe8-890c-73172763af66-kube-api-access-rzk6z podName:685a31ed-f864-4fe8-890c-73172763af66 nodeName:}" failed. No retries permitted until 2026-07-06 11:28:31.622348627 +0000 UTC m=+10.102147629 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rzk6z" (UniqueName: "kubernetes.io/projected/685a31ed-f864-4fe8-890c-73172763af66-kube-api-access-rzk6z") pod "network-check-target-5dsbt" (UID: "685a31ed-f864-4fe8-890c-73172763af66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 11:28:27.724018 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:27.723142 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6730cd7d-c083-452b-b123-e2e9f92745a1-metrics-certs\") pod \"network-metrics-daemon-rgmk8\" (UID: \"6730cd7d-c083-452b-b123-e2e9f92745a1\") " pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:28:27.724018 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:27.723242 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c4fa664-243d-4a56-91ce-a58f01d77768-original-pull-secret\") pod \"global-pull-secret-syncer-9mqf5\" (UID: \"9c4fa664-243d-4a56-91ce-a58f01d77768\") " pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:27.724018 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:27.723383 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 11:28:27.724018 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:27.723456 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c4fa664-243d-4a56-91ce-a58f01d77768-original-pull-secret podName:9c4fa664-243d-4a56-91ce-a58f01d77768 nodeName:}" failed. No retries permitted until 2026-07-06 11:28:31.723435997 +0000 UTC m=+10.203235017 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c4fa664-243d-4a56-91ce-a58f01d77768-original-pull-secret") pod "global-pull-secret-syncer-9mqf5" (UID: "9c4fa664-243d-4a56-91ce-a58f01d77768") : object "kube-system"/"original-pull-secret" not registered Jul 06 11:28:27.724018 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:27.723922 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 11:28:27.724018 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:27.723977 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6730cd7d-c083-452b-b123-e2e9f92745a1-metrics-certs podName:6730cd7d-c083-452b-b123-e2e9f92745a1 nodeName:}" failed. No retries permitted until 2026-07-06 11:28:31.723960951 +0000 UTC m=+10.203759949 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6730cd7d-c083-452b-b123-e2e9f92745a1-metrics-certs") pod "network-metrics-daemon-rgmk8" (UID: "6730cd7d-c083-452b-b123-e2e9f92745a1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 11:28:28.049127 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:28.049035 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:28:28.049627 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:28.049236 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgmk8" podUID="6730cd7d-c083-452b-b123-e2e9f92745a1" Jul 06 11:28:28.050149 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:28.049701 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:28.050149 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:28.049835 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9mqf5" podUID="9c4fa664-243d-4a56-91ce-a58f01d77768" Jul 06 11:28:28.105155 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:28.104391 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-176.ec2.internal" event={"ID":"bb7589110c767d3053aba8654c931fec","Type":"ContainerStarted","Data":"a8443d192cc19fa7975dc90aba1518deefbd4dd3b924ac99fa031eea613f737b"} Jul 06 11:28:29.046455 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:29.046418 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:28:29.046678 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:29.046561 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dsbt" podUID="685a31ed-f864-4fe8-890c-73172763af66" Jul 06 11:28:30.051478 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:30.050889 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:30.051478 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:30.051034 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9mqf5" podUID="9c4fa664-243d-4a56-91ce-a58f01d77768" Jul 06 11:28:30.051478 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:30.051126 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:28:30.051478 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:30.051255 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgmk8" podUID="6730cd7d-c083-452b-b123-e2e9f92745a1" Jul 06 11:28:31.046049 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:31.045990 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:28:31.046273 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:31.046179 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dsbt" podUID="685a31ed-f864-4fe8-890c-73172763af66" Jul 06 11:28:31.660325 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:31.660224 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzk6z\" (UniqueName: \"kubernetes.io/projected/685a31ed-f864-4fe8-890c-73172763af66-kube-api-access-rzk6z\") pod \"network-check-target-5dsbt\" (UID: \"685a31ed-f864-4fe8-890c-73172763af66\") " pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:28:31.660905 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:31.660477 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 11:28:31.660905 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:31.660506 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 11:28:31.660905 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:31.660522 2568 projected.go:196] Error preparing data for projected volume kube-api-access-rzk6z for pod openshift-network-diagnostics/network-check-target-5dsbt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 11:28:31.660905 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:31.660586 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/685a31ed-f864-4fe8-890c-73172763af66-kube-api-access-rzk6z podName:685a31ed-f864-4fe8-890c-73172763af66 nodeName:}" failed. No retries permitted until 2026-07-06 11:28:39.660565197 +0000 UTC m=+18.140364212 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rzk6z" (UniqueName: "kubernetes.io/projected/685a31ed-f864-4fe8-890c-73172763af66-kube-api-access-rzk6z") pod "network-check-target-5dsbt" (UID: "685a31ed-f864-4fe8-890c-73172763af66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 11:28:31.761821 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:31.761740 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c4fa664-243d-4a56-91ce-a58f01d77768-original-pull-secret\") pod \"global-pull-secret-syncer-9mqf5\" (UID: \"9c4fa664-243d-4a56-91ce-a58f01d77768\") " pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:31.762030 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:31.761951 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 11:28:31.762097 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:31.762035 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c4fa664-243d-4a56-91ce-a58f01d77768-original-pull-secret podName:9c4fa664-243d-4a56-91ce-a58f01d77768 nodeName:}" failed. No retries permitted until 2026-07-06 11:28:39.762011856 +0000 UTC m=+18.241810909 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c4fa664-243d-4a56-91ce-a58f01d77768-original-pull-secret") pod "global-pull-secret-syncer-9mqf5" (UID: "9c4fa664-243d-4a56-91ce-a58f01d77768") : object "kube-system"/"original-pull-secret" not registered Jul 06 11:28:31.762097 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:31.762075 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6730cd7d-c083-452b-b123-e2e9f92745a1-metrics-certs\") pod \"network-metrics-daemon-rgmk8\" (UID: \"6730cd7d-c083-452b-b123-e2e9f92745a1\") " pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:28:31.762226 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:31.762198 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 11:28:31.762279 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:31.762231 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6730cd7d-c083-452b-b123-e2e9f92745a1-metrics-certs podName:6730cd7d-c083-452b-b123-e2e9f92745a1 nodeName:}" failed. No retries permitted until 2026-07-06 11:28:39.762221822 +0000 UTC m=+18.242020817 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6730cd7d-c083-452b-b123-e2e9f92745a1-metrics-certs") pod "network-metrics-daemon-rgmk8" (UID: "6730cd7d-c083-452b-b123-e2e9f92745a1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 11:28:32.049956 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:32.049873 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:32.050106 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:32.050004 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9mqf5" podUID="9c4fa664-243d-4a56-91ce-a58f01d77768" Jul 06 11:28:32.050369 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:32.050352 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:28:32.050484 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:32.050465 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgmk8" podUID="6730cd7d-c083-452b-b123-e2e9f92745a1" Jul 06 11:28:33.046056 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:33.046016 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:28:33.046594 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:33.046152 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dsbt" podUID="685a31ed-f864-4fe8-890c-73172763af66" Jul 06 11:28:34.047084 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:34.046487 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:28:34.047084 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:34.046676 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgmk8" podUID="6730cd7d-c083-452b-b123-e2e9f92745a1" Jul 06 11:28:34.047084 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:34.046838 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:34.047084 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:34.046950 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9mqf5" podUID="9c4fa664-243d-4a56-91ce-a58f01d77768" Jul 06 11:28:35.046590 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:35.046552 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:28:35.046798 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:35.046736 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dsbt" podUID="685a31ed-f864-4fe8-890c-73172763af66" Jul 06 11:28:36.046572 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:36.046533 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:28:36.046572 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:36.046559 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:36.047106 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:36.046718 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9mqf5" podUID="9c4fa664-243d-4a56-91ce-a58f01d77768" Jul 06 11:28:36.047106 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:36.046780 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgmk8" podUID="6730cd7d-c083-452b-b123-e2e9f92745a1" Jul 06 11:28:37.046560 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:37.046471 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:28:37.046728 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:37.046576 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dsbt" podUID="685a31ed-f864-4fe8-890c-73172763af66" Jul 06 11:28:37.127735 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:37.127692 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pbv8c" event={"ID":"733213c7-49ab-4f45-a0dd-d87abe31ef28","Type":"ContainerStarted","Data":"0e9a95d0c95bafe5fe54914c34d8ad2e86d57d60b7247bc9d0ed372d56060f32"} Jul 06 11:28:37.129323 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:37.129294 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6hb" event={"ID":"c3934448-4795-4cbc-8f17-8f7e14935dd2","Type":"ContainerStarted","Data":"a5f57d7993b21e61ef54a8d6cd05a108fc63f52606507016db95775faa481d28"} Jul 06 11:28:37.145215 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:37.145168 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-176.ec2.internal" podStartSLOduration=13.14515319 podStartE2EDuration="13.14515319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 11:28:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:28:28.124893625 +0000 UTC m=+6.604692649" watchObservedRunningTime="2026-07-06 11:28:37.14515319 +0000 UTC m=+15.624952206" Jul 06 11:28:38.046610 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:38.046559 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:28:38.046795 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:38.046617 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:38.046795 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:38.046768 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgmk8" podUID="6730cd7d-c083-452b-b123-e2e9f92745a1" Jul 06 11:28:38.047327 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:38.047290 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9mqf5" podUID="9c4fa664-243d-4a56-91ce-a58f01d77768" Jul 06 11:28:38.133385 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:38.133344 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x89qj" event={"ID":"8af859eb-6aa4-41b0-89cb-8028432c8f55","Type":"ContainerStarted","Data":"72b1d1addeddc697bb3cf036f6d8a2fbb72cbc105da25f7263aebaae2c80244b"} Jul 06 11:28:38.152197 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:38.152142 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pbv8c" podStartSLOduration=3.012634692 podStartE2EDuration="15.152123925s" totalImagesPullingTime="12.139489233s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 11:28:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:28:37.145830205 +0000 UTC m=+15.625629221" watchObservedRunningTime="2026-07-06 11:28:38.152123925 +0000 UTC m=+16.631922975" Jul 06 11:28:38.152385 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:38.152364 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-x89qj" podStartSLOduration=3.053128509 podStartE2EDuration="15.152358521s" totalImagesPullingTime="12.099230012s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 11:28:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:28:38.151740543 +0000 UTC m=+16.631539573" watchObservedRunningTime="2026-07-06 11:28:38.152358521 +0000 UTC m=+16.632157537" Jul 06 11:28:39.046366 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:39.046156 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:28:39.046607 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:39.046465 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dsbt" podUID="685a31ed-f864-4fe8-890c-73172763af66" Jul 06 11:28:39.723476 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:39.723436 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzk6z\" (UniqueName: \"kubernetes.io/projected/685a31ed-f864-4fe8-890c-73172763af66-kube-api-access-rzk6z\") pod \"network-check-target-5dsbt\" (UID: \"685a31ed-f864-4fe8-890c-73172763af66\") " pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:28:39.723881 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:39.723635 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 11:28:39.723881 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:39.723656 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 11:28:39.723881 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:39.723667 2568 projected.go:196] Error preparing data for projected volume kube-api-access-rzk6z for pod openshift-network-diagnostics/network-check-target-5dsbt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 11:28:39.723881 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:39.723736 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/685a31ed-f864-4fe8-890c-73172763af66-kube-api-access-rzk6z podName:685a31ed-f864-4fe8-890c-73172763af66 nodeName:}" failed. No retries permitted until 2026-07-06 11:28:55.723720948 +0000 UTC m=+34.203519942 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rzk6z" (UniqueName: "kubernetes.io/projected/685a31ed-f864-4fe8-890c-73172763af66-kube-api-access-rzk6z") pod "network-check-target-5dsbt" (UID: "685a31ed-f864-4fe8-890c-73172763af66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 11:28:39.823954 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:39.823918 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6730cd7d-c083-452b-b123-e2e9f92745a1-metrics-certs\") pod \"network-metrics-daemon-rgmk8\" (UID: \"6730cd7d-c083-452b-b123-e2e9f92745a1\") " pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:28:39.824127 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:39.823992 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c4fa664-243d-4a56-91ce-a58f01d77768-original-pull-secret\") pod \"global-pull-secret-syncer-9mqf5\" (UID: \"9c4fa664-243d-4a56-91ce-a58f01d77768\") " pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:39.824127 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:39.824103 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 11:28:39.824127 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:39.824104 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 11:28:39.824276 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:39.824163 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c4fa664-243d-4a56-91ce-a58f01d77768-original-pull-secret podName:9c4fa664-243d-4a56-91ce-a58f01d77768 nodeName:}" failed. No retries permitted until 2026-07-06 11:28:55.824144934 +0000 UTC m=+34.303943927 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c4fa664-243d-4a56-91ce-a58f01d77768-original-pull-secret") pod "global-pull-secret-syncer-9mqf5" (UID: "9c4fa664-243d-4a56-91ce-a58f01d77768") : object "kube-system"/"original-pull-secret" not registered Jul 06 11:28:39.824276 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:39.824180 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6730cd7d-c083-452b-b123-e2e9f92745a1-metrics-certs podName:6730cd7d-c083-452b-b123-e2e9f92745a1 nodeName:}" failed. No retries permitted until 2026-07-06 11:28:55.824172944 +0000 UTC m=+34.303971939 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6730cd7d-c083-452b-b123-e2e9f92745a1-metrics-certs") pod "network-metrics-daemon-rgmk8" (UID: "6730cd7d-c083-452b-b123-e2e9f92745a1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 11:28:40.046193 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:40.046092 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:40.046356 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:40.046207 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9mqf5" podUID="9c4fa664-243d-4a56-91ce-a58f01d77768" Jul 06 11:28:40.046356 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:40.046092 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:28:40.046356 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:40.046342 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgmk8" podUID="6730cd7d-c083-452b-b123-e2e9f92745a1" Jul 06 11:28:41.046435 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:41.046391 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:28:41.046961 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:41.046528 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dsbt" podUID="685a31ed-f864-4fe8-890c-73172763af66" Jul 06 11:28:42.046637 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:42.046604 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:28:42.047205 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:42.046749 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgmk8" podUID="6730cd7d-c083-452b-b123-e2e9f92745a1" Jul 06 11:28:42.047205 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:42.046806 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:42.047205 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:42.046931 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9mqf5" podUID="9c4fa664-243d-4a56-91ce-a58f01d77768" Jul 06 11:28:43.046112 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:43.046076 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:28:43.046314 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:43.046200 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dsbt" podUID="685a31ed-f864-4fe8-890c-73172763af66" Jul 06 11:28:44.046779 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:44.046734 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:44.047379 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:44.046738 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:28:44.047379 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:44.046882 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9mqf5" podUID="9c4fa664-243d-4a56-91ce-a58f01d77768" Jul 06 11:28:44.047379 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:44.046915 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgmk8" podUID="6730cd7d-c083-452b-b123-e2e9f92745a1" Jul 06 11:28:45.046264 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:45.045933 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:28:45.046443 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:45.046344 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dsbt" podUID="685a31ed-f864-4fe8-890c-73172763af66" Jul 06 11:28:45.148110 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:45.148075 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l89bb" event={"ID":"218f64a8-8fd8-462b-972d-ff9ecedb6a74","Type":"ContainerStarted","Data":"f429440fcb704be37b26bebcdd9f4aca4a1ae352e7f0279254fcf8013665d952"} Jul 06 11:28:45.149522 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:45.149494 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pvz5d" event={"ID":"e3cd243e-9aaa-4671-b9b8-2f135c474488","Type":"ContainerStarted","Data":"110bbd3ebeb58cce3b7baa4ac1e5c9ac43315b562a36cf436093d2d3c7ac0b45"} Jul 06 11:28:45.152359 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:45.152336 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" event={"ID":"6c7bab31-dddb-4e05-a2c6-5babd5e779d4","Type":"ContainerStarted","Data":"523d7070a271018b565341e32b219b91d08a9094b49aac38847f8862666577f6"} Jul 06 11:28:45.152447 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:45.152403 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" event={"ID":"6c7bab31-dddb-4e05-a2c6-5babd5e779d4","Type":"ContainerStarted","Data":"642eff42788dc8014eca035d8ac5d2dd6a9c5aa53592585a9ae4134f77fbb38a"} Jul 06 11:28:45.152501 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:45.152449 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" event={"ID":"6c7bab31-dddb-4e05-a2c6-5babd5e779d4","Type":"ContainerStarted","Data":"ccc74b2617aae86a4138aa2fe42ee26c0cec89ade9df496b41b13bdfa0ce9fc0"} Jul 06 11:28:45.152501 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:45.152480 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" event={"ID":"6c7bab31-dddb-4e05-a2c6-5babd5e779d4","Type":"ContainerStarted","Data":"6954f48e6dbfaf182e25f8bcc52b1f0f92686ee6431bac4b52622a2a5737c07e"} Jul 06 11:28:45.152571 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:45.152509 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" event={"ID":"6c7bab31-dddb-4e05-a2c6-5babd5e779d4","Type":"ContainerStarted","Data":"05530e9344e4dc8b166055787a8e399ab652f57d729f977c2bbf4b16ddd3f231"} Jul 06 11:28:45.152571 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:45.152550 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" event={"ID":"6c7bab31-dddb-4e05-a2c6-5babd5e779d4","Type":"ContainerStarted","Data":"81cf09b825fbb5eacaa466851a9783b76216e76684f222b412ddcceba7f64301"} Jul 06 11:28:45.153661 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:45.153640 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dw874" event={"ID":"10439038-5c07-452f-bea0-a01a60ef437f","Type":"ContainerStarted","Data":"a4608ead2b854106ecaae8f8ba4466d2c85f6a4b3bb32dc69ac74c1e7d19e6fa"} Jul 06 11:28:45.155081 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:45.155060 2568 generic.go:356] "Generic (PLEG): container finished" podID="cfcd9856-b786-4cb9-b407-1fe4eafe748e" containerID="4bd4b0a70e5339536fd17493de99754b03acbc802d0f3c2ca15d24ce3c6a2462" exitCode=0 Jul 06 11:28:45.155176 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:45.155101 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-25hzg" event={"ID":"cfcd9856-b786-4cb9-b407-1fe4eafe748e","Type":"ContainerDied","Data":"4bd4b0a70e5339536fd17493de99754b03acbc802d0f3c2ca15d24ce3c6a2462"} Jul 06 11:28:45.156663 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:45.156635 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-54f75" event={"ID":"27e80ab9-5fa0-4a48-a102-95b38f491d4f","Type":"ContainerStarted","Data":"e370f8e0105b0e24e4624fb51c5e5b650dace71bae2793421d6b5087350637a2"} Jul 06 11:28:45.168959 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:45.168905 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-l89bb" podStartSLOduration=10.493394468 podStartE2EDuration="22.168887713s" totalImagesPullingTime="11.675493245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:28:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:28:45.168329016 +0000 UTC m=+23.648128033" watchObservedRunningTime="2026-07-06 11:28:45.168887713 +0000 UTC m=+23.648686729" Jul 06 11:28:45.207435 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:45.207378 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dw874" podStartSLOduration=11.648147248 podStartE2EDuration="22.207361705s" totalImagesPullingTime="10.559214457s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 11:28:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:28:45.187913584 +0000 UTC m=+23.667712600" watchObservedRunningTime="2026-07-06 11:28:45.207361705 +0000 UTC m=+23.687160721" Jul 06 11:28:45.232785 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:45.232731 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pvz5d" podStartSLOduration=2.044395628 podStartE2EDuration="22.232711495s" totalImagesPullingTime="20.188315867s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 11:28:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:28:45.207165153 +0000 UTC m=+23.686964169" watchObservedRunningTime="2026-07-06 11:28:45.232711495 +0000 UTC m=+23.712510510" Jul 06 11:28:45.251928 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:45.251875 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-54f75" podStartSLOduration=10.085268801 podStartE2EDuration="22.251857775s" totalImagesPullingTime="12.166588974s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 11:28:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:28:45.251231046 +0000 UTC m=+23.731030062" watchObservedRunningTime="2026-07-06 11:28:45.251857775 +0000 UTC m=+23.731656791" Jul 06 11:28:45.400350 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:45.400159 2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 11:28:46.008757 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:46.008629 2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T11:28:45.400339193Z","UUID":"2a6975d9-16bf-4507-a4cd-eee7d47a947a","Handler":null,"Name":"","Endpoint":""} Jul 06 11:28:46.010172 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:46.010156 2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 11:28:46.010288 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:46.010187 2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 11:28:46.045916 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:46.045881 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:28:46.046091 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:46.046009 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgmk8" podUID="6730cd7d-c083-452b-b123-e2e9f92745a1" Jul 06 11:28:46.046091 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:46.046042 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:46.046173 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:46.046111 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9mqf5" podUID="9c4fa664-243d-4a56-91ce-a58f01d77768" Jul 06 11:28:46.160940 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:46.160865 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6hb" event={"ID":"c3934448-4795-4cbc-8f17-8f7e14935dd2","Type":"ContainerStarted","Data":"2a19fc01341be4b07a5ea20c2e404c41dd6fd26a7c04006e6e7416458af65b00"} Jul 06 11:28:47.045959 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:47.045931 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:28:47.046099 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:47.046040 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dsbt" podUID="685a31ed-f864-4fe8-890c-73172763af66" Jul 06 11:28:47.165531 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:47.165493 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6hb" event={"ID":"c3934448-4795-4cbc-8f17-8f7e14935dd2","Type":"ContainerStarted","Data":"6b9a1195cde17b5cabc97c458d891d3d9121be020beda4ad9a03785591b3d18f"} Jul 06 11:28:48.045953 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:48.045916 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:28:48.046130 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:48.046070 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgmk8" podUID="6730cd7d-c083-452b-b123-e2e9f92745a1" Jul 06 11:28:48.046130 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:48.046113 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:48.046216 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:48.046199 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9mqf5" podUID="9c4fa664-243d-4a56-91ce-a58f01d77768" Jul 06 11:28:48.170901 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:48.170864 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" event={"ID":"6c7bab31-dddb-4e05-a2c6-5babd5e779d4","Type":"ContainerStarted","Data":"03ace2bbb133c4a61adc7ffd0fa849866e062f22643119f1b5d51183c5a57ace"} Jul 06 11:28:48.229651 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:48.229615 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-l89bb" Jul 06 11:28:48.230319 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:48.230294 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-l89bb" Jul 06 11:28:48.263378 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:48.263326 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dh6hb" podStartSLOduration=2.916530346 podStartE2EDuration="25.263307469s" totalImagesPullingTime="22.346777123s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 11:28:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:28:47.191441444 +0000 UTC m=+25.671240459" watchObservedRunningTime="2026-07-06 11:28:48.263307469 +0000 UTC m=+26.743106505" Jul 06 11:28:49.046670 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:49.046631 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:28:49.046859 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:49.046760 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dsbt" podUID="685a31ed-f864-4fe8-890c-73172763af66" Jul 06 11:28:49.173745 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:49.173713 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-l89bb" Jul 06 11:28:49.174247 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:49.174229 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l89bb" Jul 06 11:28:50.046544 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:50.046511 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:28:50.046753 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:50.046655 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgmk8" podUID="6730cd7d-c083-452b-b123-e2e9f92745a1" Jul 06 11:28:50.046753 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:50.046710 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:50.046876 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:50.046822 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9mqf5" podUID="9c4fa664-243d-4a56-91ce-a58f01d77768" Jul 06 11:28:51.045835 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:51.045634 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:28:51.046241 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:51.045914 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dsbt" podUID="685a31ed-f864-4fe8-890c-73172763af66" Jul 06 11:28:51.180629 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:51.180597 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" event={"ID":"6c7bab31-dddb-4e05-a2c6-5babd5e779d4","Type":"ContainerStarted","Data":"e30603365bde14dcd1f8b751a564d09bf466bcd819f0d47475d23e3bc3cf058d"} Jul 06 11:28:51.180915 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:51.180893 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:51.181166 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:51.180969 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:51.181166 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:51.181008 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:51.182778 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:51.182751 2568 generic.go:356] "Generic (PLEG): container finished" podID="cfcd9856-b786-4cb9-b407-1fe4eafe748e" containerID="db8b4f3686af75fc68bc56088ac8805b5a12a063dece7ca9c1b0cba9a858cb93" exitCode=0 Jul 06 11:28:51.182868 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:51.182828 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-25hzg" event={"ID":"cfcd9856-b786-4cb9-b407-1fe4eafe748e","Type":"ContainerDied","Data":"db8b4f3686af75fc68bc56088ac8805b5a12a063dece7ca9c1b0cba9a858cb93"} Jul 06 11:28:51.196019 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:51.195999 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:51.196147 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:51.196135 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:28:51.209392 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:51.209353 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" podStartSLOduration=7.99934057 podStartE2EDuration="28.209340229s" totalImagesPullingTime="20.209999659s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 11:28:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:28:51.207852841 +0000 UTC m=+29.687651856" watchObservedRunningTime="2026-07-06 11:28:51.209340229 +0000 UTC m=+29.689139244" Jul 06 11:28:52.047749 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:52.047714 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:28:52.047749 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:52.047748 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:52.048274 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:52.047851 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgmk8" podUID="6730cd7d-c083-452b-b123-e2e9f92745a1" Jul 06 11:28:52.048274 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:52.047981 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9mqf5" podUID="9c4fa664-243d-4a56-91ce-a58f01d77768" Jul 06 11:28:53.046694 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:53.046592 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:28:53.046837 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:53.046724 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dsbt" podUID="685a31ed-f864-4fe8-890c-73172763af66" Jul 06 11:28:53.091454 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:53.091421 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9mqf5"] Jul 06 11:28:53.091853 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:53.091545 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:53.091853 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:53.091659 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9mqf5" podUID="9c4fa664-243d-4a56-91ce-a58f01d77768" Jul 06 11:28:53.094242 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:53.094217 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rgmk8"] Jul 06 11:28:53.094382 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:53.094320 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:28:53.094449 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:53.094427 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgmk8" podUID="6730cd7d-c083-452b-b123-e2e9f92745a1" Jul 06 11:28:53.097374 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:53.097351 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5dsbt"] Jul 06 11:28:53.190488 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:53.190453 2568 generic.go:356] "Generic (PLEG): container finished" podID="cfcd9856-b786-4cb9-b407-1fe4eafe748e" containerID="ee78f447ee77719dec6cb078485feab54e1b41e696e9d7d35b25d4868d2df291" exitCode=0 Jul 06 11:28:53.190699 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:53.190542 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:28:53.190699 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:53.190539 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-25hzg" event={"ID":"cfcd9856-b786-4cb9-b407-1fe4eafe748e","Type":"ContainerDied","Data":"ee78f447ee77719dec6cb078485feab54e1b41e696e9d7d35b25d4868d2df291"} Jul 06 11:28:53.190699 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:53.190660 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dsbt" podUID="685a31ed-f864-4fe8-890c-73172763af66" Jul 06 11:28:55.045751 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:55.045714 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:55.045751 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:55.045744 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:28:55.046200 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:55.045750 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:28:55.046200 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:55.045813 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9mqf5" podUID="9c4fa664-243d-4a56-91ce-a58f01d77768" Jul 06 11:28:55.046200 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:55.045886 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dsbt" podUID="685a31ed-f864-4fe8-890c-73172763af66" Jul 06 11:28:55.046200 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:55.045954 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgmk8" podUID="6730cd7d-c083-452b-b123-e2e9f92745a1" Jul 06 11:28:55.198378 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:55.198346 2568 generic.go:356] "Generic (PLEG): container finished" podID="cfcd9856-b786-4cb9-b407-1fe4eafe748e" containerID="bec3bbdf7548a1cd02d1994e210de090630efbc87637e21af0c70e3ef307570f" exitCode=0 Jul 06 11:28:55.198527 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:55.198410 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-25hzg" event={"ID":"cfcd9856-b786-4cb9-b407-1fe4eafe748e","Type":"ContainerDied","Data":"bec3bbdf7548a1cd02d1994e210de090630efbc87637e21af0c70e3ef307570f"} Jul 06 11:28:55.738953 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:55.738920 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzk6z\" (UniqueName: \"kubernetes.io/projected/685a31ed-f864-4fe8-890c-73172763af66-kube-api-access-rzk6z\") pod \"network-check-target-5dsbt\" (UID: \"685a31ed-f864-4fe8-890c-73172763af66\") " pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:28:55.739141 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:55.739080 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 11:28:55.739141 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:55.739096 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 11:28:55.739141 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:55.739105 2568 projected.go:196] Error preparing data for projected volume kube-api-access-rzk6z for pod openshift-network-diagnostics/network-check-target-5dsbt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 11:28:55.739255 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:55.739158 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/685a31ed-f864-4fe8-890c-73172763af66-kube-api-access-rzk6z podName:685a31ed-f864-4fe8-890c-73172763af66 nodeName:}" failed. No retries permitted until 2026-07-06 11:29:27.739144389 +0000 UTC m=+66.218943382 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rzk6z" (UniqueName: "kubernetes.io/projected/685a31ed-f864-4fe8-890c-73172763af66-kube-api-access-rzk6z") pod "network-check-target-5dsbt" (UID: "685a31ed-f864-4fe8-890c-73172763af66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 11:28:55.839631 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:55.839596 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6730cd7d-c083-452b-b123-e2e9f92745a1-metrics-certs\") pod \"network-metrics-daemon-rgmk8\" (UID: \"6730cd7d-c083-452b-b123-e2e9f92745a1\") " pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:28:55.839834 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:55.839722 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c4fa664-243d-4a56-91ce-a58f01d77768-original-pull-secret\") pod \"global-pull-secret-syncer-9mqf5\" (UID: \"9c4fa664-243d-4a56-91ce-a58f01d77768\") " pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:55.839834 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:55.839787 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 11:28:55.839834 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:55.839814 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 11:28:55.839953 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:55.839850 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6730cd7d-c083-452b-b123-e2e9f92745a1-metrics-certs podName:6730cd7d-c083-452b-b123-e2e9f92745a1 nodeName:}" failed. No retries permitted until 2026-07-06 11:29:27.839830755 +0000 UTC m=+66.319629772 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6730cd7d-c083-452b-b123-e2e9f92745a1-metrics-certs") pod "network-metrics-daemon-rgmk8" (UID: "6730cd7d-c083-452b-b123-e2e9f92745a1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 11:28:55.839953 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:55.839865 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c4fa664-243d-4a56-91ce-a58f01d77768-original-pull-secret podName:9c4fa664-243d-4a56-91ce-a58f01d77768 nodeName:}" failed. No retries permitted until 2026-07-06 11:29:27.839858694 +0000 UTC m=+66.319657688 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c4fa664-243d-4a56-91ce-a58f01d77768-original-pull-secret") pod "global-pull-secret-syncer-9mqf5" (UID: "9c4fa664-243d-4a56-91ce-a58f01d77768") : object "kube-system"/"original-pull-secret" not registered Jul 06 11:28:57.045764 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.045724 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:57.045764 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.045764 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:28:57.046236 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.045742 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:28:57.046236 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:57.045832 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9mqf5" podUID="9c4fa664-243d-4a56-91ce-a58f01d77768" Jul 06 11:28:57.046236 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:57.045893 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5dsbt" podUID="685a31ed-f864-4fe8-890c-73172763af66" Jul 06 11:28:57.046236 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:57.045976 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rgmk8" podUID="6730cd7d-c083-452b-b123-e2e9f92745a1" Jul 06 11:28:57.357869 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.357791 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-176.ec2.internal" event="NodeReady" Jul 06 11:28:57.358035 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.357923 2568 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 11:28:57.411814 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.411781 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-577cf8786b-5q4rz"] Jul 06 11:28:57.414907 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.414883 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:28:57.420487 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.420462 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 11:28:57.420697 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.420651 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 11:28:57.421331 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.421309 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 11:28:57.423787 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.423765 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rbs4n\"" Jul 06 11:28:57.429833 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.429812 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 11:28:57.436334 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.436313 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-577cf8786b-5q4rz"] Jul 06 11:28:57.451610 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.451577 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-image-registry-private-configuration\") pod \"image-registry-577cf8786b-5q4rz\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:28:57.451776 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.451621 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-ca-trust-extracted\") pod \"image-registry-577cf8786b-5q4rz\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:28:57.451776 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.451654 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-certificates\") pod \"image-registry-577cf8786b-5q4rz\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:28:57.451776 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.451700 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-installation-pull-secrets\") pod \"image-registry-577cf8786b-5q4rz\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:28:57.451776 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.451754 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-bound-sa-token\") pod \"image-registry-577cf8786b-5q4rz\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:28:57.451981 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.451866 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq6fr\" (UniqueName: \"kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-kube-api-access-kq6fr\") pod \"image-registry-577cf8786b-5q4rz\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:28:57.451981 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.451902 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-trusted-ca\") pod \"image-registry-577cf8786b-5q4rz\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:28:57.451981 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.451928 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-tls\") pod \"image-registry-577cf8786b-5q4rz\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:28:57.452513 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.452491 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-khjfx"] Jul 06 11:28:57.455362 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.455343 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dvbz4"] Jul 06 11:28:57.455505 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.455486 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-khjfx" Jul 06 11:28:57.458240 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.458218 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dvbz4" Jul 06 11:28:57.458853 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.458833 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 11:28:57.458951 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.458911 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-989xb\"" Jul 06 11:28:57.459010 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.459001 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 11:28:57.459223 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.459201 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 11:28:57.461120 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.461100 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 11:28:57.461227 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.461154 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bpbrx\"" Jul 06 11:28:57.461360 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.461344 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 11:28:57.465753 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.465729 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-khjfx"] Jul 06 11:28:57.468472 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.468449 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dvbz4"] Jul 06 11:28:57.552527 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.552489 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-image-registry-private-configuration\") pod \"image-registry-577cf8786b-5q4rz\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:28:57.552527 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.552529 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-ca-trust-extracted\") pod \"image-registry-577cf8786b-5q4rz\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:28:57.552826 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.552552 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-certificates\") pod \"image-registry-577cf8786b-5q4rz\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:28:57.552826 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.552573 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e130dede-7e5f-4cd5-8a39-d47ad7f48a2c-cert\") pod \"ingress-canary-khjfx\" (UID: \"e130dede-7e5f-4cd5-8a39-d47ad7f48a2c\") " pod="openshift-ingress-canary/ingress-canary-khjfx" Jul 06 11:28:57.552826 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.552599 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-installation-pull-secrets\") pod \"image-registry-577cf8786b-5q4rz\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:28:57.552826 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.552701 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-bound-sa-token\") pod \"image-registry-577cf8786b-5q4rz\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:28:57.553016 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.552874 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5zc2\" (UniqueName: \"kubernetes.io/projected/e130dede-7e5f-4cd5-8a39-d47ad7f48a2c-kube-api-access-d5zc2\") pod \"ingress-canary-khjfx\" (UID: \"e130dede-7e5f-4cd5-8a39-d47ad7f48a2c\") " pod="openshift-ingress-canary/ingress-canary-khjfx" Jul 06 11:28:57.553151 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.553130 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-ca-trust-extracted\") pod \"image-registry-577cf8786b-5q4rz\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:28:57.553414 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.552975 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq6fr\" (UniqueName: \"kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-kube-api-access-kq6fr\") pod \"image-registry-577cf8786b-5q4rz\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:28:57.553485 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.553434 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-trusted-ca\") pod \"image-registry-577cf8786b-5q4rz\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:28:57.553485 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.553462 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-tls\") pod \"image-registry-577cf8786b-5q4rz\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:28:57.553578 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:57.553554 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 11:28:57.553578 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:57.553568 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-577cf8786b-5q4rz: secret "image-registry-tls" not found Jul 06 11:28:57.557169 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.554042 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-certificates\") pod \"image-registry-577cf8786b-5q4rz\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:28:57.557169 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:57.554042 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-tls podName:d435fc8e-6b2f-4f9c-94c8-be96d31f65a5 nodeName:}" failed. No retries permitted until 2026-07-06 11:28:58.054015305 +0000 UTC m=+36.533814319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-tls") pod "image-registry-577cf8786b-5q4rz" (UID: "d435fc8e-6b2f-4f9c-94c8-be96d31f65a5") : secret "image-registry-tls" not found Jul 06 11:28:57.558379 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.558336 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-installation-pull-secrets\") pod \"image-registry-577cf8786b-5q4rz\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:28:57.558379 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.558338 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-image-registry-private-configuration\") pod \"image-registry-577cf8786b-5q4rz\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:28:57.558567 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.558539 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-trusted-ca\") pod \"image-registry-577cf8786b-5q4rz\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:28:57.561995 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.561972 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq6fr\" (UniqueName: \"kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-kube-api-access-kq6fr\") pod \"image-registry-577cf8786b-5q4rz\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:28:57.563510 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.563486 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-bound-sa-token\") pod \"image-registry-577cf8786b-5q4rz\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:28:57.654629 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.654592 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5zc2\" (UniqueName: \"kubernetes.io/projected/e130dede-7e5f-4cd5-8a39-d47ad7f48a2c-kube-api-access-d5zc2\") pod \"ingress-canary-khjfx\" (UID: \"e130dede-7e5f-4cd5-8a39-d47ad7f48a2c\") " pod="openshift-ingress-canary/ingress-canary-khjfx" Jul 06 11:28:57.654818 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.654696 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03ee7031-c88a-47a2-b37c-0349a1787567-tmp-dir\") pod \"dns-default-dvbz4\" (UID: \"03ee7031-c88a-47a2-b37c-0349a1787567\") " pod="openshift-dns/dns-default-dvbz4" Jul 06 11:28:57.654818 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.654736 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/03ee7031-c88a-47a2-b37c-0349a1787567-config-volume\") pod \"dns-default-dvbz4\" (UID: \"03ee7031-c88a-47a2-b37c-0349a1787567\") " pod="openshift-dns/dns-default-dvbz4" Jul 06 11:28:57.654818 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.654769 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03ee7031-c88a-47a2-b37c-0349a1787567-metrics-tls\") pod \"dns-default-dvbz4\" (UID: \"03ee7031-c88a-47a2-b37c-0349a1787567\") " pod="openshift-dns/dns-default-dvbz4" Jul 06 11:28:57.654944 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.654824 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e130dede-7e5f-4cd5-8a39-d47ad7f48a2c-cert\") pod \"ingress-canary-khjfx\" (UID: \"e130dede-7e5f-4cd5-8a39-d47ad7f48a2c\") " pod="openshift-ingress-canary/ingress-canary-khjfx" Jul 06 11:28:57.654944 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.654846 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7skr\" (UniqueName: \"kubernetes.io/projected/03ee7031-c88a-47a2-b37c-0349a1787567-kube-api-access-l7skr\") pod \"dns-default-dvbz4\" (UID: \"03ee7031-c88a-47a2-b37c-0349a1787567\") " pod="openshift-dns/dns-default-dvbz4" Jul 06 11:28:57.657203 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.657179 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e130dede-7e5f-4cd5-8a39-d47ad7f48a2c-cert\") pod \"ingress-canary-khjfx\" (UID: \"e130dede-7e5f-4cd5-8a39-d47ad7f48a2c\") " pod="openshift-ingress-canary/ingress-canary-khjfx" Jul 06 11:28:57.663830 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.663812 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5zc2\" (UniqueName: \"kubernetes.io/projected/e130dede-7e5f-4cd5-8a39-d47ad7f48a2c-kube-api-access-d5zc2\") pod \"ingress-canary-khjfx\" (UID: \"e130dede-7e5f-4cd5-8a39-d47ad7f48a2c\") " pod="openshift-ingress-canary/ingress-canary-khjfx" Jul 06 11:28:57.755398 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.755367 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03ee7031-c88a-47a2-b37c-0349a1787567-tmp-dir\") pod \"dns-default-dvbz4\" (UID: \"03ee7031-c88a-47a2-b37c-0349a1787567\") " pod="openshift-dns/dns-default-dvbz4" Jul 06 11:28:57.755630 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.755426 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/03ee7031-c88a-47a2-b37c-0349a1787567-config-volume\") pod \"dns-default-dvbz4\" (UID: \"03ee7031-c88a-47a2-b37c-0349a1787567\") " pod="openshift-dns/dns-default-dvbz4" Jul 06 11:28:57.755630 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.755453 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03ee7031-c88a-47a2-b37c-0349a1787567-metrics-tls\") pod \"dns-default-dvbz4\" (UID: \"03ee7031-c88a-47a2-b37c-0349a1787567\") " pod="openshift-dns/dns-default-dvbz4" Jul 06 11:28:57.755630 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.755477 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7skr\" (UniqueName: \"kubernetes.io/projected/03ee7031-c88a-47a2-b37c-0349a1787567-kube-api-access-l7skr\") pod \"dns-default-dvbz4\" (UID: \"03ee7031-c88a-47a2-b37c-0349a1787567\") " pod="openshift-dns/dns-default-dvbz4" Jul 06 11:28:57.755630 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:57.755607 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 11:28:57.755903 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:57.755695 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03ee7031-c88a-47a2-b37c-0349a1787567-metrics-tls podName:03ee7031-c88a-47a2-b37c-0349a1787567 nodeName:}" failed. No retries permitted until 2026-07-06 11:28:58.255661846 +0000 UTC m=+36.735460841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/03ee7031-c88a-47a2-b37c-0349a1787567-metrics-tls") pod "dns-default-dvbz4" (UID: "03ee7031-c88a-47a2-b37c-0349a1787567") : secret "dns-default-metrics-tls" not found Jul 06 11:28:57.755903 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.755709 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03ee7031-c88a-47a2-b37c-0349a1787567-tmp-dir\") pod \"dns-default-dvbz4\" (UID: \"03ee7031-c88a-47a2-b37c-0349a1787567\") " pod="openshift-dns/dns-default-dvbz4" Jul 06 11:28:57.756118 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.756096 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/03ee7031-c88a-47a2-b37c-0349a1787567-config-volume\") pod \"dns-default-dvbz4\" (UID: \"03ee7031-c88a-47a2-b37c-0349a1787567\") " pod="openshift-dns/dns-default-dvbz4" Jul 06 11:28:57.766801 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.766776 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7skr\" (UniqueName: \"kubernetes.io/projected/03ee7031-c88a-47a2-b37c-0349a1787567-kube-api-access-l7skr\") pod \"dns-default-dvbz4\" (UID: \"03ee7031-c88a-47a2-b37c-0349a1787567\") " pod="openshift-dns/dns-default-dvbz4" Jul 06 11:28:57.767567 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.767546 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-khjfx" Jul 06 11:28:57.921189 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:57.921107 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-khjfx"] Jul 06 11:28:57.925830 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:28:57.925655 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode130dede_7e5f_4cd5_8a39_d47ad7f48a2c.slice/crio-9e0980e17df708d7a7d6e3c2fa0cf45a89590a0b64808f3d30deb4480a6eb565 WatchSource:0}: Error finding container 9e0980e17df708d7a7d6e3c2fa0cf45a89590a0b64808f3d30deb4480a6eb565: Status 404 returned error can't find the container with id 9e0980e17df708d7a7d6e3c2fa0cf45a89590a0b64808f3d30deb4480a6eb565 Jul 06 11:28:58.058344 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:58.058316 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-tls\") pod \"image-registry-577cf8786b-5q4rz\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:28:58.059137 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:58.058445 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 11:28:58.059137 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:58.058461 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-577cf8786b-5q4rz: secret "image-registry-tls" not found Jul 06 11:28:58.059137 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:58.058618 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-tls podName:d435fc8e-6b2f-4f9c-94c8-be96d31f65a5 nodeName:}" failed. No retries permitted until 2026-07-06 11:28:59.058498519 +0000 UTC m=+37.538297533 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-tls") pod "image-registry-577cf8786b-5q4rz" (UID: "d435fc8e-6b2f-4f9c-94c8-be96d31f65a5") : secret "image-registry-tls" not found Jul 06 11:28:58.208019 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:58.207933 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-khjfx" event={"ID":"e130dede-7e5f-4cd5-8a39-d47ad7f48a2c","Type":"ContainerStarted","Data":"9e0980e17df708d7a7d6e3c2fa0cf45a89590a0b64808f3d30deb4480a6eb565"} Jul 06 11:28:58.260715 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:58.260665 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03ee7031-c88a-47a2-b37c-0349a1787567-metrics-tls\") pod \"dns-default-dvbz4\" (UID: \"03ee7031-c88a-47a2-b37c-0349a1787567\") " pod="openshift-dns/dns-default-dvbz4" Jul 06 11:28:58.260902 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:58.260838 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 11:28:58.260959 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:58.260919 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03ee7031-c88a-47a2-b37c-0349a1787567-metrics-tls podName:03ee7031-c88a-47a2-b37c-0349a1787567 nodeName:}" failed. No retries permitted until 2026-07-06 11:28:59.260897966 +0000 UTC m=+37.740696977 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/03ee7031-c88a-47a2-b37c-0349a1787567-metrics-tls") pod "dns-default-dvbz4" (UID: "03ee7031-c88a-47a2-b37c-0349a1787567") : secret "dns-default-metrics-tls" not found Jul 06 11:28:59.046262 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:59.046224 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:28:59.046464 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:59.046224 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:28:59.046464 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:59.046224 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:28:59.051508 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:59.051477 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 11:28:59.051814 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:59.051793 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w64mh\"" Jul 06 11:28:59.052006 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:59.051991 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 11:28:59.052180 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:59.052161 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 11:28:59.052180 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:59.052175 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 11:28:59.052387 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:59.052178 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kdds4\"" Jul 06 11:28:59.069431 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:59.069405 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-tls\") pod \"image-registry-577cf8786b-5q4rz\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:28:59.069828 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:59.069569 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 11:28:59.069828 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:59.069583 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-577cf8786b-5q4rz: secret "image-registry-tls" not found Jul 06 11:28:59.069828 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:59.069641 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-tls podName:d435fc8e-6b2f-4f9c-94c8-be96d31f65a5 nodeName:}" failed. No retries permitted until 2026-07-06 11:29:01.069621983 +0000 UTC m=+39.549420995 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-tls") pod "image-registry-577cf8786b-5q4rz" (UID: "d435fc8e-6b2f-4f9c-94c8-be96d31f65a5") : secret "image-registry-tls" not found Jul 06 11:28:59.271668 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:28:59.271627 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03ee7031-c88a-47a2-b37c-0349a1787567-metrics-tls\") pod \"dns-default-dvbz4\" (UID: \"03ee7031-c88a-47a2-b37c-0349a1787567\") " pod="openshift-dns/dns-default-dvbz4" Jul 06 11:28:59.271881 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:59.271806 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 11:28:59.271951 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:28:59.271882 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03ee7031-c88a-47a2-b37c-0349a1787567-metrics-tls podName:03ee7031-c88a-47a2-b37c-0349a1787567 nodeName:}" failed. No retries permitted until 2026-07-06 11:29:01.271860933 +0000 UTC m=+39.751659942 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/03ee7031-c88a-47a2-b37c-0349a1787567-metrics-tls") pod "dns-default-dvbz4" (UID: "03ee7031-c88a-47a2-b37c-0349a1787567") : secret "dns-default-metrics-tls" not found Jul 06 11:29:01.085639 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:01.085601 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-tls\") pod \"image-registry-577cf8786b-5q4rz\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:29:01.086123 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:01.085826 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 11:29:01.086123 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:01.085853 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-577cf8786b-5q4rz: secret "image-registry-tls" not found Jul 06 11:29:01.086123 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:01.085936 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-tls podName:d435fc8e-6b2f-4f9c-94c8-be96d31f65a5 nodeName:}" failed. No retries permitted until 2026-07-06 11:29:05.08591364 +0000 UTC m=+43.565712646 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-tls") pod "image-registry-577cf8786b-5q4rz" (UID: "d435fc8e-6b2f-4f9c-94c8-be96d31f65a5") : secret "image-registry-tls" not found Jul 06 11:29:01.218714 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:01.218603 2568 generic.go:356] "Generic (PLEG): container finished" podID="cfcd9856-b786-4cb9-b407-1fe4eafe748e" containerID="b217d3b5cad2a094c78282f991687b1db8dc3e0246aced575d96471fd8b978e3" exitCode=0 Jul 06 11:29:01.218714 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:01.218695 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-25hzg" event={"ID":"cfcd9856-b786-4cb9-b407-1fe4eafe748e","Type":"ContainerDied","Data":"b217d3b5cad2a094c78282f991687b1db8dc3e0246aced575d96471fd8b978e3"} Jul 06 11:29:01.220329 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:01.220298 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-khjfx" event={"ID":"e130dede-7e5f-4cd5-8a39-d47ad7f48a2c","Type":"ContainerStarted","Data":"e54ae8284906694dbfbebbd62caad4b1c43273b3eed42a6853987da651b86f45"} Jul 06 11:29:01.272252 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:01.271992 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-khjfx" podStartSLOduration=1.284358166 podStartE2EDuration="4.271969154s" totalImagesPullingTime="2.987610988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:28:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:29:01.271345289 +0000 UTC m=+39.751144306" watchObservedRunningTime="2026-07-06 11:29:01.271969154 +0000 UTC m=+39.751768171" Jul 06 11:29:01.287262 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:01.287069 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03ee7031-c88a-47a2-b37c-0349a1787567-metrics-tls\") pod \"dns-default-dvbz4\" (UID: \"03ee7031-c88a-47a2-b37c-0349a1787567\") " pod="openshift-dns/dns-default-dvbz4" Jul 06 11:29:01.287262 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:01.287234 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 11:29:01.287502 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:01.287303 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03ee7031-c88a-47a2-b37c-0349a1787567-metrics-tls podName:03ee7031-c88a-47a2-b37c-0349a1787567 nodeName:}" failed. No retries permitted until 2026-07-06 11:29:05.28728441 +0000 UTC m=+43.767083418 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/03ee7031-c88a-47a2-b37c-0349a1787567-metrics-tls") pod "dns-default-dvbz4" (UID: "03ee7031-c88a-47a2-b37c-0349a1787567") : secret "dns-default-metrics-tls" not found Jul 06 11:29:02.225671 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:02.225630 2568 generic.go:356] "Generic (PLEG): container finished" podID="cfcd9856-b786-4cb9-b407-1fe4eafe748e" containerID="bd6e1e4e07202cd2dc55dd4aa54a850c102408cfd5d8960b492f3a1af41b6e05" exitCode=0 Jul 06 11:29:02.226076 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:02.225717 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-25hzg" event={"ID":"cfcd9856-b786-4cb9-b407-1fe4eafe748e","Type":"ContainerDied","Data":"bd6e1e4e07202cd2dc55dd4aa54a850c102408cfd5d8960b492f3a1af41b6e05"} Jul 06 11:29:03.231517 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:03.231474 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-25hzg" event={"ID":"cfcd9856-b786-4cb9-b407-1fe4eafe748e","Type":"ContainerStarted","Data":"fffa2207d4c2a8f5e5cbc77903adebc6c8626928a52478c56a73cb77bf9e7177"} Jul 06 11:29:03.259772 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:03.259719 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-25hzg" podStartSLOduration=15.530812199 podStartE2EDuration="40.259703046s" totalImagesPullingTime="24.728890847s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 11:28:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:29:03.258293705 +0000 UTC m=+41.738092722" watchObservedRunningTime="2026-07-06 11:29:03.259703046 +0000 UTC m=+41.739502056" Jul 06 11:29:05.115174 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:05.115127 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-tls\") pod \"image-registry-577cf8786b-5q4rz\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:29:05.115574 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:05.115278 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 11:29:05.115574 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:05.115297 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-577cf8786b-5q4rz: secret "image-registry-tls" not found Jul 06 11:29:05.115574 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:05.115352 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-tls podName:d435fc8e-6b2f-4f9c-94c8-be96d31f65a5 nodeName:}" failed. No retries permitted until 2026-07-06 11:29:13.115336488 +0000 UTC m=+51.595135485 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-tls") pod "image-registry-577cf8786b-5q4rz" (UID: "d435fc8e-6b2f-4f9c-94c8-be96d31f65a5") : secret "image-registry-tls" not found Jul 06 11:29:05.317408 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:05.317366 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03ee7031-c88a-47a2-b37c-0349a1787567-metrics-tls\") pod \"dns-default-dvbz4\" (UID: \"03ee7031-c88a-47a2-b37c-0349a1787567\") " pod="openshift-dns/dns-default-dvbz4" Jul 06 11:29:05.317586 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:05.317523 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 11:29:05.317628 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:05.317586 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03ee7031-c88a-47a2-b37c-0349a1787567-metrics-tls podName:03ee7031-c88a-47a2-b37c-0349a1787567 nodeName:}" failed. No retries permitted until 2026-07-06 11:29:13.317569936 +0000 UTC m=+51.797368942 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/03ee7031-c88a-47a2-b37c-0349a1787567-metrics-tls") pod "dns-default-dvbz4" (UID: "03ee7031-c88a-47a2-b37c-0349a1787567") : secret "dns-default-metrics-tls" not found Jul 06 11:29:13.179991 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:13.179942 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-tls\") pod \"image-registry-577cf8786b-5q4rz\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:29:13.180401 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:13.180093 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 11:29:13.180401 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:13.180114 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-577cf8786b-5q4rz: secret "image-registry-tls" not found Jul 06 11:29:13.180401 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:13.180172 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-tls podName:d435fc8e-6b2f-4f9c-94c8-be96d31f65a5 nodeName:}" failed. No retries permitted until 2026-07-06 11:29:29.180155132 +0000 UTC m=+67.659954130 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-tls") pod "image-registry-577cf8786b-5q4rz" (UID: "d435fc8e-6b2f-4f9c-94c8-be96d31f65a5") : secret "image-registry-tls" not found Jul 06 11:29:13.380978 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:13.380941 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03ee7031-c88a-47a2-b37c-0349a1787567-metrics-tls\") pod \"dns-default-dvbz4\" (UID: \"03ee7031-c88a-47a2-b37c-0349a1787567\") " pod="openshift-dns/dns-default-dvbz4" Jul 06 11:29:13.381170 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:13.381099 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 11:29:13.381208 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:13.381177 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03ee7031-c88a-47a2-b37c-0349a1787567-metrics-tls podName:03ee7031-c88a-47a2-b37c-0349a1787567 nodeName:}" failed. No retries permitted until 2026-07-06 11:29:29.381161742 +0000 UTC m=+67.860960739 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/03ee7031-c88a-47a2-b37c-0349a1787567-metrics-tls") pod "dns-default-dvbz4" (UID: "03ee7031-c88a-47a2-b37c-0349a1787567") : secret "dns-default-metrics-tls" not found Jul 06 11:29:20.844835 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.844791 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5cq2k"] Jul 06 11:29:20.846938 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.846907 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5cq2k" Jul 06 11:29:20.848547 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.848518 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6knbf"] Jul 06 11:29:20.850366 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.850348 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jjrll"] Jul 06 11:29:20.850487 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.850474 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6knbf" Jul 06 11:29:20.851138 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.851114 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 06 11:29:20.851309 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.851287 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 06 11:29:20.851426 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.851286 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 06 11:29:20.851490 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.851449 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-cswbf\"" Jul 06 11:29:20.852274 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.852251 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 06 11:29:20.852488 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.852268 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-gsnkd"] Jul 06 11:29:20.852488 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.852377 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jjrll" Jul 06 11:29:20.852913 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.852884 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 06 11:29:20.854152 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.854137 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-gsnkd" Jul 06 11:29:20.854924 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.854905 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 06 11:29:20.855179 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.855162 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-jtz6q\"" Jul 06 11:29:20.855281 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.855215 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 06 11:29:20.855343 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.855311 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 06 11:29:20.856773 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.856748 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 06 11:29:20.856773 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.856763 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 11:29:20.856925 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.856775 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-w5cz8\"" Jul 06 11:29:20.856925 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.856828 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 11:29:20.856925 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.856778 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 11:29:20.857940 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.857924 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 11:29:20.857940 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.857933 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 06 11:29:20.858073 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.857965 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-5dnhv\"" Jul 06 11:29:20.861191 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.861169 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 06 11:29:20.864801 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.864781 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 06 11:29:20.865421 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.865397 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 06 11:29:20.871059 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.871038 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5cq2k"] Jul 06 11:29:20.872347 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.872285 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jjrll"] Jul 06 11:29:20.874085 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.874062 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-gsnkd"] Jul 06 11:29:20.882515 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.882494 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6knbf"] Jul 06 11:29:20.940885 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.940856 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bgwx\" (UniqueName: \"kubernetes.io/projected/eba80240-4cdd-4537-a223-616b0196c115-kube-api-access-9bgwx\") pod \"service-ca-operator-5d46d9ffd4-6knbf\" (UID: \"eba80240-4cdd-4537-a223-616b0196c115\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6knbf" Jul 06 11:29:20.940885 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.940887 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a12983e-eac8-4b8b-8baf-688b9f7c4907-serving-cert\") pod \"insights-operator-5c847dcfc4-gsnkd\" (UID: \"8a12983e-eac8-4b8b-8baf-688b9f7c4907\") " pod="openshift-insights/insights-operator-5c847dcfc4-gsnkd" Jul 06 11:29:20.941096 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.940918 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n72jz\" (UniqueName: \"kubernetes.io/projected/8a12983e-eac8-4b8b-8baf-688b9f7c4907-kube-api-access-n72jz\") pod \"insights-operator-5c847dcfc4-gsnkd\" (UID: \"8a12983e-eac8-4b8b-8baf-688b9f7c4907\") " pod="openshift-insights/insights-operator-5c847dcfc4-gsnkd" Jul 06 11:29:20.941096 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.940938 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2864a14b-0f18-45e1-8c00-129a0c26570c-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-5cq2k\" (UID: \"2864a14b-0f18-45e1-8c00-129a0c26570c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5cq2k" Jul 06 11:29:20.941096 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.940985 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a12983e-eac8-4b8b-8baf-688b9f7c4907-tmp\") pod \"insights-operator-5c847dcfc4-gsnkd\" (UID: \"8a12983e-eac8-4b8b-8baf-688b9f7c4907\") " pod="openshift-insights/insights-operator-5c847dcfc4-gsnkd" Jul 06 11:29:20.941096 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.941024 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a12983e-eac8-4b8b-8baf-688b9f7c4907-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-gsnkd\" (UID: \"8a12983e-eac8-4b8b-8baf-688b9f7c4907\") " pod="openshift-insights/insights-operator-5c847dcfc4-gsnkd" Jul 06 11:29:20.941096 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.941050 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eba80240-4cdd-4537-a223-616b0196c115-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-6knbf\" (UID: \"eba80240-4cdd-4537-a223-616b0196c115\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6knbf" Jul 06 11:29:20.941096 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.941071 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2864a14b-0f18-45e1-8c00-129a0c26570c-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-5cq2k\" (UID: \"2864a14b-0f18-45e1-8c00-129a0c26570c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5cq2k" Jul 06 11:29:20.941278 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.941112 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a12983e-eac8-4b8b-8baf-688b9f7c4907-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-gsnkd\" (UID: \"8a12983e-eac8-4b8b-8baf-688b9f7c4907\") " pod="openshift-insights/insights-operator-5c847dcfc4-gsnkd" Jul 06 11:29:20.941278 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.941140 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbpgk\" (UniqueName: \"kubernetes.io/projected/2864a14b-0f18-45e1-8c00-129a0c26570c-kube-api-access-fbpgk\") pod \"kube-storage-version-migrator-operator-846bf668f7-5cq2k\" (UID: \"2864a14b-0f18-45e1-8c00-129a0c26570c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5cq2k" Jul 06 11:29:20.941278 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.941162 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e16731d-a528-49c9-8998-dcc1d0b44957-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-jjrll\" (UID: \"6e16731d-a528-49c9-8998-dcc1d0b44957\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jjrll" Jul 06 11:29:20.941278 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.941196 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8a12983e-eac8-4b8b-8baf-688b9f7c4907-snapshots\") pod \"insights-operator-5c847dcfc4-gsnkd\" (UID: \"8a12983e-eac8-4b8b-8baf-688b9f7c4907\") " pod="openshift-insights/insights-operator-5c847dcfc4-gsnkd" Jul 06 11:29:20.941278 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.941238 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpsdw\" (UniqueName: \"kubernetes.io/projected/6e16731d-a528-49c9-8998-dcc1d0b44957-kube-api-access-bpsdw\") pod \"cluster-monitoring-operator-64786d9cbb-jjrll\" (UID: \"6e16731d-a528-49c9-8998-dcc1d0b44957\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jjrll" Jul 06 11:29:20.941278 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.941265 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6e16731d-a528-49c9-8998-dcc1d0b44957-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-jjrll\" (UID: \"6e16731d-a528-49c9-8998-dcc1d0b44957\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jjrll" Jul 06 11:29:20.941457 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:20.941298 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eba80240-4cdd-4537-a223-616b0196c115-config\") pod \"service-ca-operator-5d46d9ffd4-6knbf\" (UID: \"eba80240-4cdd-4537-a223-616b0196c115\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6knbf" Jul 06 11:29:21.041958 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.041919 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bgwx\" (UniqueName: \"kubernetes.io/projected/eba80240-4cdd-4537-a223-616b0196c115-kube-api-access-9bgwx\") pod \"service-ca-operator-5d46d9ffd4-6knbf\" (UID: \"eba80240-4cdd-4537-a223-616b0196c115\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6knbf" Jul 06 11:29:21.041958 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.041957 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a12983e-eac8-4b8b-8baf-688b9f7c4907-serving-cert\") pod \"insights-operator-5c847dcfc4-gsnkd\" (UID: \"8a12983e-eac8-4b8b-8baf-688b9f7c4907\") " pod="openshift-insights/insights-operator-5c847dcfc4-gsnkd" Jul 06 11:29:21.042211 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.041991 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n72jz\" (UniqueName: \"kubernetes.io/projected/8a12983e-eac8-4b8b-8baf-688b9f7c4907-kube-api-access-n72jz\") pod \"insights-operator-5c847dcfc4-gsnkd\" (UID: \"8a12983e-eac8-4b8b-8baf-688b9f7c4907\") " pod="openshift-insights/insights-operator-5c847dcfc4-gsnkd" Jul 06 11:29:21.042211 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.042020 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2864a14b-0f18-45e1-8c00-129a0c26570c-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-5cq2k\" (UID: \"2864a14b-0f18-45e1-8c00-129a0c26570c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5cq2k" Jul 06 11:29:21.042211 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.042042 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a12983e-eac8-4b8b-8baf-688b9f7c4907-tmp\") pod \"insights-operator-5c847dcfc4-gsnkd\" (UID: \"8a12983e-eac8-4b8b-8baf-688b9f7c4907\") " pod="openshift-insights/insights-operator-5c847dcfc4-gsnkd" Jul 06 11:29:21.042211 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.042062 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a12983e-eac8-4b8b-8baf-688b9f7c4907-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-gsnkd\" (UID: \"8a12983e-eac8-4b8b-8baf-688b9f7c4907\") " pod="openshift-insights/insights-operator-5c847dcfc4-gsnkd" Jul 06 11:29:21.042211 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.042084 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eba80240-4cdd-4537-a223-616b0196c115-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-6knbf\" (UID: \"eba80240-4cdd-4537-a223-616b0196c115\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6knbf" Jul 06 11:29:21.042211 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.042107 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2864a14b-0f18-45e1-8c00-129a0c26570c-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-5cq2k\" (UID: \"2864a14b-0f18-45e1-8c00-129a0c26570c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5cq2k" Jul 06 11:29:21.042656 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.042632 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a12983e-eac8-4b8b-8baf-688b9f7c4907-tmp\") pod \"insights-operator-5c847dcfc4-gsnkd\" (UID: \"8a12983e-eac8-4b8b-8baf-688b9f7c4907\") " pod="openshift-insights/insights-operator-5c847dcfc4-gsnkd" Jul 06 11:29:21.042836 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.042812 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a12983e-eac8-4b8b-8baf-688b9f7c4907-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-gsnkd\" (UID: \"8a12983e-eac8-4b8b-8baf-688b9f7c4907\") " pod="openshift-insights/insights-operator-5c847dcfc4-gsnkd" Jul 06 11:29:21.042893 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.042837 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2864a14b-0f18-45e1-8c00-129a0c26570c-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-5cq2k\" (UID: \"2864a14b-0f18-45e1-8c00-129a0c26570c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5cq2k" Jul 06 11:29:21.042893 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.042868 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbpgk\" (UniqueName: \"kubernetes.io/projected/2864a14b-0f18-45e1-8c00-129a0c26570c-kube-api-access-fbpgk\") pod \"kube-storage-version-migrator-operator-846bf668f7-5cq2k\" (UID: \"2864a14b-0f18-45e1-8c00-129a0c26570c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5cq2k" Jul 06 11:29:21.042976 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.042903 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e16731d-a528-49c9-8998-dcc1d0b44957-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-jjrll\" (UID: \"6e16731d-a528-49c9-8998-dcc1d0b44957\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jjrll" Jul 06 11:29:21.042976 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.042927 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8a12983e-eac8-4b8b-8baf-688b9f7c4907-snapshots\") pod \"insights-operator-5c847dcfc4-gsnkd\" (UID: \"8a12983e-eac8-4b8b-8baf-688b9f7c4907\") " pod="openshift-insights/insights-operator-5c847dcfc4-gsnkd" Jul 06 11:29:21.042976 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.042960 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpsdw\" (UniqueName: \"kubernetes.io/projected/6e16731d-a528-49c9-8998-dcc1d0b44957-kube-api-access-bpsdw\") pod \"cluster-monitoring-operator-64786d9cbb-jjrll\" (UID: \"6e16731d-a528-49c9-8998-dcc1d0b44957\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jjrll" Jul 06 11:29:21.043103 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.042989 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6e16731d-a528-49c9-8998-dcc1d0b44957-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-jjrll\" (UID: \"6e16731d-a528-49c9-8998-dcc1d0b44957\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jjrll" Jul 06 11:29:21.043103 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.043011 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eba80240-4cdd-4537-a223-616b0196c115-config\") pod \"service-ca-operator-5d46d9ffd4-6knbf\" (UID: \"eba80240-4cdd-4537-a223-616b0196c115\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6knbf" Jul 06 11:29:21.043190 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.043131 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a12983e-eac8-4b8b-8baf-688b9f7c4907-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-gsnkd\" (UID: \"8a12983e-eac8-4b8b-8baf-688b9f7c4907\") " pod="openshift-insights/insights-operator-5c847dcfc4-gsnkd" Jul 06 11:29:21.043273 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:21.043240 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 11:29:21.043349 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:21.043321 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e16731d-a528-49c9-8998-dcc1d0b44957-cluster-monitoring-operator-tls podName:6e16731d-a528-49c9-8998-dcc1d0b44957 nodeName:}" failed. No retries permitted until 2026-07-06 11:29:21.543302219 +0000 UTC m=+60.023101231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6e16731d-a528-49c9-8998-dcc1d0b44957-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-jjrll" (UID: "6e16731d-a528-49c9-8998-dcc1d0b44957") : secret "cluster-monitoring-operator-tls" not found Jul 06 11:29:21.043532 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.043513 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eba80240-4cdd-4537-a223-616b0196c115-config\") pod \"service-ca-operator-5d46d9ffd4-6knbf\" (UID: \"eba80240-4cdd-4537-a223-616b0196c115\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6knbf" Jul 06 11:29:21.043599 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.043557 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8a12983e-eac8-4b8b-8baf-688b9f7c4907-snapshots\") pod \"insights-operator-5c847dcfc4-gsnkd\" (UID: \"8a12983e-eac8-4b8b-8baf-688b9f7c4907\") " pod="openshift-insights/insights-operator-5c847dcfc4-gsnkd" Jul 06 11:29:21.043662 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.043644 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a12983e-eac8-4b8b-8baf-688b9f7c4907-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-gsnkd\" (UID: \"8a12983e-eac8-4b8b-8baf-688b9f7c4907\") " pod="openshift-insights/insights-operator-5c847dcfc4-gsnkd" Jul 06 11:29:21.044259 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.044235 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6e16731d-a528-49c9-8998-dcc1d0b44957-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-jjrll\" (UID: \"6e16731d-a528-49c9-8998-dcc1d0b44957\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jjrll" Jul 06 11:29:21.046056 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.046032 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eba80240-4cdd-4537-a223-616b0196c115-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-6knbf\" (UID: \"eba80240-4cdd-4537-a223-616b0196c115\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6knbf" Jul 06 11:29:21.046137 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.046071 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2864a14b-0f18-45e1-8c00-129a0c26570c-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-5cq2k\" (UID: \"2864a14b-0f18-45e1-8c00-129a0c26570c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5cq2k" Jul 06 11:29:21.046238 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.046219 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a12983e-eac8-4b8b-8baf-688b9f7c4907-serving-cert\") pod \"insights-operator-5c847dcfc4-gsnkd\" (UID: \"8a12983e-eac8-4b8b-8baf-688b9f7c4907\") " pod="openshift-insights/insights-operator-5c847dcfc4-gsnkd" Jul 06 11:29:21.070908 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.070888 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbpgk\" (UniqueName: \"kubernetes.io/projected/2864a14b-0f18-45e1-8c00-129a0c26570c-kube-api-access-fbpgk\") pod \"kube-storage-version-migrator-operator-846bf668f7-5cq2k\" (UID: \"2864a14b-0f18-45e1-8c00-129a0c26570c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5cq2k" Jul 06 11:29:21.072406 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.072386 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpsdw\" (UniqueName: \"kubernetes.io/projected/6e16731d-a528-49c9-8998-dcc1d0b44957-kube-api-access-bpsdw\") pod \"cluster-monitoring-operator-64786d9cbb-jjrll\" (UID: \"6e16731d-a528-49c9-8998-dcc1d0b44957\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jjrll" Jul 06 11:29:21.074774 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.074755 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bgwx\" (UniqueName: \"kubernetes.io/projected/eba80240-4cdd-4537-a223-616b0196c115-kube-api-access-9bgwx\") pod \"service-ca-operator-5d46d9ffd4-6knbf\" (UID: \"eba80240-4cdd-4537-a223-616b0196c115\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6knbf" Jul 06 11:29:21.080236 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.080215 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n72jz\" (UniqueName: \"kubernetes.io/projected/8a12983e-eac8-4b8b-8baf-688b9f7c4907-kube-api-access-n72jz\") pod \"insights-operator-5c847dcfc4-gsnkd\" (UID: \"8a12983e-eac8-4b8b-8baf-688b9f7c4907\") " pod="openshift-insights/insights-operator-5c847dcfc4-gsnkd" Jul 06 11:29:21.158632 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.158600 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5cq2k" Jul 06 11:29:21.165303 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.165277 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6knbf" Jul 06 11:29:21.180473 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.180449 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-gsnkd" Jul 06 11:29:21.322106 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.322072 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5cq2k"] Jul 06 11:29:21.324996 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:29:21.324963 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2864a14b_0f18_45e1_8c00_129a0c26570c.slice/crio-838537d1177b12c9e0ea26b2dd2b57259897254fd5d4d966b30fa49b1f91c0cf WatchSource:0}: Error finding container 838537d1177b12c9e0ea26b2dd2b57259897254fd5d4d966b30fa49b1f91c0cf: Status 404 returned error can't find the container with id 838537d1177b12c9e0ea26b2dd2b57259897254fd5d4d966b30fa49b1f91c0cf Jul 06 11:29:21.336132 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.336010 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6knbf"] Jul 06 11:29:21.340733 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:29:21.340707 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba80240_4cdd_4537_a223_616b0196c115.slice/crio-d26912a1dc6d229f0e80a29d63d05f1da5ece811e3cf622d9335f73f02b7056f WatchSource:0}: Error finding container d26912a1dc6d229f0e80a29d63d05f1da5ece811e3cf622d9335f73f02b7056f: Status 404 returned error can't find the container with id d26912a1dc6d229f0e80a29d63d05f1da5ece811e3cf622d9335f73f02b7056f Jul 06 11:29:21.352640 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.352613 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-gsnkd"] Jul 06 11:29:21.356631 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:29:21.356604 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a12983e_eac8_4b8b_8baf_688b9f7c4907.slice/crio-7d0c53413b21d87c42788981b958546008abe6734ccf6d77a8b6e8729fb9dc0c WatchSource:0}: Error finding container 7d0c53413b21d87c42788981b958546008abe6734ccf6d77a8b6e8729fb9dc0c: Status 404 returned error can't find the container with id 7d0c53413b21d87c42788981b958546008abe6734ccf6d77a8b6e8729fb9dc0c Jul 06 11:29:21.546951 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:21.546834 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e16731d-a528-49c9-8998-dcc1d0b44957-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-jjrll\" (UID: \"6e16731d-a528-49c9-8998-dcc1d0b44957\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jjrll" Jul 06 11:29:21.547112 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:21.547000 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 11:29:21.547112 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:21.547070 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e16731d-a528-49c9-8998-dcc1d0b44957-cluster-monitoring-operator-tls podName:6e16731d-a528-49c9-8998-dcc1d0b44957 nodeName:}" failed. No retries permitted until 2026-07-06 11:29:22.547054177 +0000 UTC m=+61.026853171 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6e16731d-a528-49c9-8998-dcc1d0b44957-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-jjrll" (UID: "6e16731d-a528-49c9-8998-dcc1d0b44957") : secret "cluster-monitoring-operator-tls" not found Jul 06 11:29:22.281729 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:22.281660 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-gsnkd" event={"ID":"8a12983e-eac8-4b8b-8baf-688b9f7c4907","Type":"ContainerStarted","Data":"7d0c53413b21d87c42788981b958546008abe6734ccf6d77a8b6e8729fb9dc0c"} Jul 06 11:29:22.284336 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:22.284264 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6knbf" event={"ID":"eba80240-4cdd-4537-a223-616b0196c115","Type":"ContainerStarted","Data":"d26912a1dc6d229f0e80a29d63d05f1da5ece811e3cf622d9335f73f02b7056f"} Jul 06 11:29:22.286345 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:22.286316 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5cq2k" event={"ID":"2864a14b-0f18-45e1-8c00-129a0c26570c","Type":"ContainerStarted","Data":"838537d1177b12c9e0ea26b2dd2b57259897254fd5d4d966b30fa49b1f91c0cf"} Jul 06 11:29:22.554033 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:22.553933 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e16731d-a528-49c9-8998-dcc1d0b44957-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-jjrll\" (UID: \"6e16731d-a528-49c9-8998-dcc1d0b44957\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jjrll" Jul 06 11:29:22.554206 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:22.554113 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 11:29:22.554206 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:22.554185 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e16731d-a528-49c9-8998-dcc1d0b44957-cluster-monitoring-operator-tls podName:6e16731d-a528-49c9-8998-dcc1d0b44957 nodeName:}" failed. No retries permitted until 2026-07-06 11:29:24.55416314 +0000 UTC m=+63.033962147 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6e16731d-a528-49c9-8998-dcc1d0b44957-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-jjrll" (UID: "6e16731d-a528-49c9-8998-dcc1d0b44957") : secret "cluster-monitoring-operator-tls" not found Jul 06 11:29:23.209040 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:23.209012 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tn42c" Jul 06 11:29:24.572550 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:24.572522 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e16731d-a528-49c9-8998-dcc1d0b44957-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-jjrll\" (UID: \"6e16731d-a528-49c9-8998-dcc1d0b44957\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jjrll" Jul 06 11:29:24.572853 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:24.572666 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 11:29:24.572853 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:24.572747 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e16731d-a528-49c9-8998-dcc1d0b44957-cluster-monitoring-operator-tls podName:6e16731d-a528-49c9-8998-dcc1d0b44957 nodeName:}" failed. No retries permitted until 2026-07-06 11:29:28.572731405 +0000 UTC m=+67.052530398 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6e16731d-a528-49c9-8998-dcc1d0b44957-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-jjrll" (UID: "6e16731d-a528-49c9-8998-dcc1d0b44957") : secret "cluster-monitoring-operator-tls" not found Jul 06 11:29:25.299191 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:25.299146 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6knbf" event={"ID":"eba80240-4cdd-4537-a223-616b0196c115","Type":"ContainerStarted","Data":"d3c49e80605d644c6b456d9417a8aaea4ddb4b468fc2ed358e84838182268285"} Jul 06 11:29:25.301106 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:25.301064 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5cq2k" event={"ID":"2864a14b-0f18-45e1-8c00-129a0c26570c","Type":"ContainerStarted","Data":"ca4d22619fbe9f388b9b1a3e5fe8b1ba63f610511183ca9cc78535519a3b67b9"} Jul 06 11:29:25.302721 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:25.302645 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-gsnkd" event={"ID":"8a12983e-eac8-4b8b-8baf-688b9f7c4907","Type":"ContainerStarted","Data":"f1a9463088605d9bf10c266789a4a2763978d18100d2e18cecf84f4ef9f2d27e"} Jul 06 11:29:25.335417 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:25.335356 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6knbf" podStartSLOduration=2.088617111 podStartE2EDuration="5.335334347s" totalImagesPullingTime="3.246717236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:29:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:29:25.318635431 +0000 UTC m=+63.798434450" watchObservedRunningTime="2026-07-06 11:29:25.335334347 +0000 UTC m=+63.815133361" Jul 06 11:29:25.336462 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:25.336423 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-gsnkd" podStartSLOduration=2.111623137 podStartE2EDuration="5.336406206s" totalImagesPullingTime="3.224783069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:29:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:29:25.33489808 +0000 UTC m=+63.814697096" watchObservedRunningTime="2026-07-06 11:29:25.336406206 +0000 UTC m=+63.816205224" Jul 06 11:29:25.660192 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:25.660128 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5cq2k" podStartSLOduration=2.401779067 podStartE2EDuration="5.660105057s" totalImagesPullingTime="3.25832599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:29:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:29:25.351801079 +0000 UTC m=+63.831600095" watchObservedRunningTime="2026-07-06 11:29:25.660105057 +0000 UTC m=+64.139904074" Jul 06 11:29:25.662210 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:25.662184 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tgp62"] Jul 06 11:29:25.666449 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:25.666424 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tgp62" Jul 06 11:29:25.669971 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:25.669947 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 11:29:25.670728 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:25.670552 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hcchr\"" Jul 06 11:29:25.670852 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:25.670435 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 11:29:25.680597 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:25.680574 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tgp62"] Jul 06 11:29:25.784233 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:25.784193 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eddc93f7-ea09-405e-9488-7c7e5f710ac2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tgp62\" (UID: \"eddc93f7-ea09-405e-9488-7c7e5f710ac2\") " pod="openshift-insights/insights-runtime-extractor-tgp62" Jul 06 11:29:25.784415 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:25.784250 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eddc93f7-ea09-405e-9488-7c7e5f710ac2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tgp62\" (UID: \"eddc93f7-ea09-405e-9488-7c7e5f710ac2\") " pod="openshift-insights/insights-runtime-extractor-tgp62" Jul 06 11:29:25.784415 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:25.784387 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eddc93f7-ea09-405e-9488-7c7e5f710ac2-crio-socket\") pod \"insights-runtime-extractor-tgp62\" (UID: \"eddc93f7-ea09-405e-9488-7c7e5f710ac2\") " pod="openshift-insights/insights-runtime-extractor-tgp62" Jul 06 11:29:25.784512 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:25.784423 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eddc93f7-ea09-405e-9488-7c7e5f710ac2-data-volume\") pod \"insights-runtime-extractor-tgp62\" (UID: \"eddc93f7-ea09-405e-9488-7c7e5f710ac2\") " pod="openshift-insights/insights-runtime-extractor-tgp62" Jul 06 11:29:25.784512 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:25.784443 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jtc4\" (UniqueName: \"kubernetes.io/projected/eddc93f7-ea09-405e-9488-7c7e5f710ac2-kube-api-access-6jtc4\") pod \"insights-runtime-extractor-tgp62\" (UID: \"eddc93f7-ea09-405e-9488-7c7e5f710ac2\") " pod="openshift-insights/insights-runtime-extractor-tgp62" Jul 06 11:29:25.885312 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:25.885271 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eddc93f7-ea09-405e-9488-7c7e5f710ac2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tgp62\" (UID: \"eddc93f7-ea09-405e-9488-7c7e5f710ac2\") " pod="openshift-insights/insights-runtime-extractor-tgp62" Jul 06 11:29:25.885487 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:25.885390 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eddc93f7-ea09-405e-9488-7c7e5f710ac2-crio-socket\") pod \"insights-runtime-extractor-tgp62\" (UID: \"eddc93f7-ea09-405e-9488-7c7e5f710ac2\") " pod="openshift-insights/insights-runtime-extractor-tgp62" Jul 06 11:29:25.885487 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:25.885428 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eddc93f7-ea09-405e-9488-7c7e5f710ac2-data-volume\") pod \"insights-runtime-extractor-tgp62\" (UID: \"eddc93f7-ea09-405e-9488-7c7e5f710ac2\") " pod="openshift-insights/insights-runtime-extractor-tgp62" Jul 06 11:29:25.885487 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:25.885435 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 11:29:25.885487 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:25.885455 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jtc4\" (UniqueName: \"kubernetes.io/projected/eddc93f7-ea09-405e-9488-7c7e5f710ac2-kube-api-access-6jtc4\") pod \"insights-runtime-extractor-tgp62\" (UID: \"eddc93f7-ea09-405e-9488-7c7e5f710ac2\") " pod="openshift-insights/insights-runtime-extractor-tgp62" Jul 06 11:29:25.885657 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:25.885511 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eddc93f7-ea09-405e-9488-7c7e5f710ac2-insights-runtime-extractor-tls podName:eddc93f7-ea09-405e-9488-7c7e5f710ac2 nodeName:}" failed. No retries permitted until 2026-07-06 11:29:26.385490289 +0000 UTC m=+64.865289304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eddc93f7-ea09-405e-9488-7c7e5f710ac2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tgp62" (UID: "eddc93f7-ea09-405e-9488-7c7e5f710ac2") : secret "insights-runtime-extractor-tls" not found Jul 06 11:29:25.885657 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:25.885529 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eddc93f7-ea09-405e-9488-7c7e5f710ac2-crio-socket\") pod \"insights-runtime-extractor-tgp62\" (UID: \"eddc93f7-ea09-405e-9488-7c7e5f710ac2\") " pod="openshift-insights/insights-runtime-extractor-tgp62" Jul 06 11:29:25.885657 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:25.885558 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eddc93f7-ea09-405e-9488-7c7e5f710ac2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tgp62\" (UID: \"eddc93f7-ea09-405e-9488-7c7e5f710ac2\") " pod="openshift-insights/insights-runtime-extractor-tgp62" Jul 06 11:29:25.885657 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:25.885639 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 11:29:25.885953 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:25.885708 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eddc93f7-ea09-405e-9488-7c7e5f710ac2-kube-rbac-proxy-cm podName:eddc93f7-ea09-405e-9488-7c7e5f710ac2 nodeName:}" failed. No retries permitted until 2026-07-06 11:29:26.385690466 +0000 UTC m=+64.865489475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eddc93f7-ea09-405e-9488-7c7e5f710ac2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tgp62" (UID: "eddc93f7-ea09-405e-9488-7c7e5f710ac2") : configmap "kube-rbac-proxy" not found Jul 06 11:29:25.885953 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:25.885805 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eddc93f7-ea09-405e-9488-7c7e5f710ac2-data-volume\") pod \"insights-runtime-extractor-tgp62\" (UID: \"eddc93f7-ea09-405e-9488-7c7e5f710ac2\") " pod="openshift-insights/insights-runtime-extractor-tgp62" Jul 06 11:29:25.895057 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:25.895034 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jtc4\" (UniqueName: \"kubernetes.io/projected/eddc93f7-ea09-405e-9488-7c7e5f710ac2-kube-api-access-6jtc4\") pod \"insights-runtime-extractor-tgp62\" (UID: \"eddc93f7-ea09-405e-9488-7c7e5f710ac2\") " pod="openshift-insights/insights-runtime-extractor-tgp62" Jul 06 11:29:26.389505 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:26.389473 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eddc93f7-ea09-405e-9488-7c7e5f710ac2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tgp62\" (UID: \"eddc93f7-ea09-405e-9488-7c7e5f710ac2\") " pod="openshift-insights/insights-runtime-extractor-tgp62" Jul 06 11:29:26.389721 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:26.389536 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eddc93f7-ea09-405e-9488-7c7e5f710ac2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tgp62\" (UID: \"eddc93f7-ea09-405e-9488-7c7e5f710ac2\") " pod="openshift-insights/insights-runtime-extractor-tgp62" Jul 06 11:29:26.389721 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:26.389595 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 11:29:26.389721 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:26.389612 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 11:29:26.389721 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:26.389662 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eddc93f7-ea09-405e-9488-7c7e5f710ac2-insights-runtime-extractor-tls podName:eddc93f7-ea09-405e-9488-7c7e5f710ac2 nodeName:}" failed. No retries permitted until 2026-07-06 11:29:27.38964692 +0000 UTC m=+65.869445917 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eddc93f7-ea09-405e-9488-7c7e5f710ac2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tgp62" (UID: "eddc93f7-ea09-405e-9488-7c7e5f710ac2") : secret "insights-runtime-extractor-tls" not found Jul 06 11:29:26.389721 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:26.389697 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eddc93f7-ea09-405e-9488-7c7e5f710ac2-kube-rbac-proxy-cm podName:eddc93f7-ea09-405e-9488-7c7e5f710ac2 nodeName:}" failed. No retries permitted until 2026-07-06 11:29:27.389668325 +0000 UTC m=+65.869467319 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eddc93f7-ea09-405e-9488-7c7e5f710ac2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tgp62" (UID: "eddc93f7-ea09-405e-9488-7c7e5f710ac2") : configmap "kube-rbac-proxy" not found Jul 06 11:29:27.399437 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:27.399397 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eddc93f7-ea09-405e-9488-7c7e5f710ac2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tgp62\" (UID: \"eddc93f7-ea09-405e-9488-7c7e5f710ac2\") " pod="openshift-insights/insights-runtime-extractor-tgp62" Jul 06 11:29:27.399874 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:27.399447 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eddc93f7-ea09-405e-9488-7c7e5f710ac2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tgp62\" (UID: \"eddc93f7-ea09-405e-9488-7c7e5f710ac2\") " pod="openshift-insights/insights-runtime-extractor-tgp62" Jul 06 11:29:27.399874 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:27.399528 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 11:29:27.399874 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:27.399546 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 11:29:27.399874 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:27.399594 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eddc93f7-ea09-405e-9488-7c7e5f710ac2-kube-rbac-proxy-cm podName:eddc93f7-ea09-405e-9488-7c7e5f710ac2 nodeName:}" failed. No retries permitted until 2026-07-06 11:29:29.39958024 +0000 UTC m=+67.879379255 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eddc93f7-ea09-405e-9488-7c7e5f710ac2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tgp62" (UID: "eddc93f7-ea09-405e-9488-7c7e5f710ac2") : configmap "kube-rbac-proxy" not found Jul 06 11:29:27.399874 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:27.399608 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eddc93f7-ea09-405e-9488-7c7e5f710ac2-insights-runtime-extractor-tls podName:eddc93f7-ea09-405e-9488-7c7e5f710ac2 nodeName:}" failed. No retries permitted until 2026-07-06 11:29:29.39960205 +0000 UTC m=+67.879401044 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eddc93f7-ea09-405e-9488-7c7e5f710ac2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tgp62" (UID: "eddc93f7-ea09-405e-9488-7c7e5f710ac2") : secret "insights-runtime-extractor-tls" not found Jul 06 11:29:27.666389 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:27.666310 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dw874_10439038-5c07-452f-bea0-a01a60ef437f/dns-node-resolver/0.log" Jul 06 11:29:27.802692 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:27.802653 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzk6z\" (UniqueName: \"kubernetes.io/projected/685a31ed-f864-4fe8-890c-73172763af66-kube-api-access-rzk6z\") pod \"network-check-target-5dsbt\" (UID: \"685a31ed-f864-4fe8-890c-73172763af66\") " pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:29:27.808080 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:27.808060 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 11:29:27.816330 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:27.816306 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 11:29:27.826219 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:27.826197 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzk6z\" (UniqueName: \"kubernetes.io/projected/685a31ed-f864-4fe8-890c-73172763af66-kube-api-access-rzk6z\") pod \"network-check-target-5dsbt\" (UID: \"685a31ed-f864-4fe8-890c-73172763af66\") " pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:29:27.864209 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:27.864188 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kdds4\"" Jul 06 11:29:27.871631 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:27.871598 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:29:27.904240 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:27.904202 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6730cd7d-c083-452b-b123-e2e9f92745a1-metrics-certs\") pod \"network-metrics-daemon-rgmk8\" (UID: \"6730cd7d-c083-452b-b123-e2e9f92745a1\") " pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:29:27.904386 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:27.904361 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c4fa664-243d-4a56-91ce-a58f01d77768-original-pull-secret\") pod \"global-pull-secret-syncer-9mqf5\" (UID: \"9c4fa664-243d-4a56-91ce-a58f01d77768\") " pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:29:27.908313 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:27.908296 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 11:29:27.908759 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:27.908740 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 11:29:27.915153 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:27.915123 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 06 11:29:27.915298 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:27.915235 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6730cd7d-c083-452b-b123-e2e9f92745a1-metrics-certs podName:6730cd7d-c083-452b-b123-e2e9f92745a1 nodeName:}" failed. No retries permitted until 2026-07-06 11:30:31.915210967 +0000 UTC m=+130.395009981 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6730cd7d-c083-452b-b123-e2e9f92745a1-metrics-certs") pod "network-metrics-daemon-rgmk8" (UID: "6730cd7d-c083-452b-b123-e2e9f92745a1") : secret "metrics-daemon-secret" not found Jul 06 11:29:27.917497 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:27.917452 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c4fa664-243d-4a56-91ce-a58f01d77768-original-pull-secret\") pod \"global-pull-secret-syncer-9mqf5\" (UID: \"9c4fa664-243d-4a56-91ce-a58f01d77768\") " pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:29:28.035840 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:28.035814 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5dsbt"] Jul 06 11:29:28.036844 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:29:28.036812 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod685a31ed_f864_4fe8_890c_73172763af66.slice/crio-e06a4ae4637ca991413642ed3669adc15191b9335fd0bc76bb1e5ddce611d887 WatchSource:0}: Error finding container e06a4ae4637ca991413642ed3669adc15191b9335fd0bc76bb1e5ddce611d887: Status 404 returned error can't find the container with id e06a4ae4637ca991413642ed3669adc15191b9335fd0bc76bb1e5ddce611d887 Jul 06 11:29:28.176075 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:28.175972 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9mqf5" Jul 06 11:29:28.307814 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:28.307780 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9mqf5"] Jul 06 11:29:28.311289 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:28.311256 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5dsbt" event={"ID":"685a31ed-f864-4fe8-890c-73172763af66","Type":"ContainerStarted","Data":"e06a4ae4637ca991413642ed3669adc15191b9335fd0bc76bb1e5ddce611d887"} Jul 06 11:29:28.311788 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:29:28.311768 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c4fa664_243d_4a56_91ce_a58f01d77768.slice/crio-4d51e305f0cb04069beb970f2ef862e5e968c18daad5090f58a8c1cf2bd74895 WatchSource:0}: Error finding container 4d51e305f0cb04069beb970f2ef862e5e968c18daad5090f58a8c1cf2bd74895: Status 404 returned error can't find the container with id 4d51e305f0cb04069beb970f2ef862e5e968c18daad5090f58a8c1cf2bd74895 Jul 06 11:29:28.609714 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:28.609605 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e16731d-a528-49c9-8998-dcc1d0b44957-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-jjrll\" (UID: \"6e16731d-a528-49c9-8998-dcc1d0b44957\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jjrll" Jul 06 11:29:28.610160 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:28.609772 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 11:29:28.610160 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:28.609847 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e16731d-a528-49c9-8998-dcc1d0b44957-cluster-monitoring-operator-tls podName:6e16731d-a528-49c9-8998-dcc1d0b44957 nodeName:}" failed. No retries permitted until 2026-07-06 11:29:36.609826623 +0000 UTC m=+75.089625631 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6e16731d-a528-49c9-8998-dcc1d0b44957-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-jjrll" (UID: "6e16731d-a528-49c9-8998-dcc1d0b44957") : secret "cluster-monitoring-operator-tls" not found Jul 06 11:29:28.667824 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:28.667791 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pbv8c_733213c7-49ab-4f45-a0dd-d87abe31ef28/node-ca/0.log" Jul 06 11:29:29.215233 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:29.215188 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-tls\") pod \"image-registry-577cf8786b-5q4rz\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:29:29.215415 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:29.215356 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 11:29:29.215415 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:29.215371 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-577cf8786b-5q4rz: secret "image-registry-tls" not found Jul 06 11:29:29.215589 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:29.215435 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-tls podName:d435fc8e-6b2f-4f9c-94c8-be96d31f65a5 nodeName:}" failed. No retries permitted until 2026-07-06 11:30:01.215421441 +0000 UTC m=+99.695220454 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-tls") pod "image-registry-577cf8786b-5q4rz" (UID: "d435fc8e-6b2f-4f9c-94c8-be96d31f65a5") : secret "image-registry-tls" not found Jul 06 11:29:29.265962 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:29.265927 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-khjfx_e130dede-7e5f-4cd5-8a39-d47ad7f48a2c/serve-healthcheck-canary/0.log" Jul 06 11:29:29.316722 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:29.316665 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9mqf5" event={"ID":"9c4fa664-243d-4a56-91ce-a58f01d77768","Type":"ContainerStarted","Data":"4d51e305f0cb04069beb970f2ef862e5e968c18daad5090f58a8c1cf2bd74895"} Jul 06 11:29:29.416460 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:29.416402 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eddc93f7-ea09-405e-9488-7c7e5f710ac2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tgp62\" (UID: \"eddc93f7-ea09-405e-9488-7c7e5f710ac2\") " pod="openshift-insights/insights-runtime-extractor-tgp62" Jul 06 11:29:29.416460 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:29.416445 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03ee7031-c88a-47a2-b37c-0349a1787567-metrics-tls\") pod \"dns-default-dvbz4\" (UID: \"03ee7031-c88a-47a2-b37c-0349a1787567\") " pod="openshift-dns/dns-default-dvbz4" Jul 06 11:29:29.416460 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:29.416465 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eddc93f7-ea09-405e-9488-7c7e5f710ac2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tgp62\" (UID: \"eddc93f7-ea09-405e-9488-7c7e5f710ac2\") " pod="openshift-insights/insights-runtime-extractor-tgp62" Jul 06 11:29:29.416460 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:29.416533 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 11:29:29.416460 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:29.416584 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 11:29:29.416460 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:29.416592 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 11:29:29.417857 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:29.416605 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eddc93f7-ea09-405e-9488-7c7e5f710ac2-kube-rbac-proxy-cm podName:eddc93f7-ea09-405e-9488-7c7e5f710ac2 nodeName:}" failed. No retries permitted until 2026-07-06 11:29:33.41658846 +0000 UTC m=+71.896387469 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eddc93f7-ea09-405e-9488-7c7e5f710ac2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tgp62" (UID: "eddc93f7-ea09-405e-9488-7c7e5f710ac2") : configmap "kube-rbac-proxy" not found Jul 06 11:29:29.417857 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:29.416637 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eddc93f7-ea09-405e-9488-7c7e5f710ac2-insights-runtime-extractor-tls podName:eddc93f7-ea09-405e-9488-7c7e5f710ac2 nodeName:}" failed. No retries permitted until 2026-07-06 11:29:33.416623243 +0000 UTC m=+71.896422243 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eddc93f7-ea09-405e-9488-7c7e5f710ac2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tgp62" (UID: "eddc93f7-ea09-405e-9488-7c7e5f710ac2") : secret "insights-runtime-extractor-tls" not found Jul 06 11:29:29.417857 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:29.416653 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03ee7031-c88a-47a2-b37c-0349a1787567-metrics-tls podName:03ee7031-c88a-47a2-b37c-0349a1787567 nodeName:}" failed. No retries permitted until 2026-07-06 11:30:01.416645337 +0000 UTC m=+99.896444333 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/03ee7031-c88a-47a2-b37c-0349a1787567-metrics-tls") pod "dns-default-dvbz4" (UID: "03ee7031-c88a-47a2-b37c-0349a1787567") : secret "dns-default-metrics-tls" not found Jul 06 11:29:33.331428 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:33.331320 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9mqf5" event={"ID":"9c4fa664-243d-4a56-91ce-a58f01d77768","Type":"ContainerStarted","Data":"5596203ee15b4fad9c4ac23c961a4199a9397dd36193f10eec911bd564ace3bc"} Jul 06 11:29:33.332835 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:33.332813 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5dsbt" event={"ID":"685a31ed-f864-4fe8-890c-73172763af66","Type":"ContainerStarted","Data":"fb6fd8f14b790c58206a6753402794edb8968c3b8b2783938b84f9242c7c1c1c"} Jul 06 11:29:33.332959 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:33.332946 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:29:33.349367 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:33.349324 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9mqf5" podStartSLOduration=65.591929455 podStartE2EDuration="1m10.349305235s" totalImagesPullingTime="4.75737578s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 11:28:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:29:33.348830783 +0000 UTC m=+71.828629802" watchObservedRunningTime="2026-07-06 11:29:33.349305235 +0000 UTC m=+71.829104252" Jul 06 11:29:33.365714 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:33.365644 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5dsbt" podStartSLOduration=65.34760397 podStartE2EDuration="1m10.365629418s" totalImagesPullingTime="5.018025448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:28:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:29:33.365475292 +0000 UTC m=+71.845274305" watchObservedRunningTime="2026-07-06 11:29:33.365629418 +0000 UTC m=+71.845428433" Jul 06 11:29:33.455259 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:33.455216 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eddc93f7-ea09-405e-9488-7c7e5f710ac2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tgp62\" (UID: \"eddc93f7-ea09-405e-9488-7c7e5f710ac2\") " pod="openshift-insights/insights-runtime-extractor-tgp62" Jul 06 11:29:33.455457 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:33.455275 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eddc93f7-ea09-405e-9488-7c7e5f710ac2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tgp62\" (UID: \"eddc93f7-ea09-405e-9488-7c7e5f710ac2\") " pod="openshift-insights/insights-runtime-extractor-tgp62" Jul 06 11:29:33.455457 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:33.455369 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 11:29:33.455457 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:33.455369 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 11:29:33.455457 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:33.455436 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eddc93f7-ea09-405e-9488-7c7e5f710ac2-insights-runtime-extractor-tls podName:eddc93f7-ea09-405e-9488-7c7e5f710ac2 nodeName:}" failed. No retries permitted until 2026-07-06 11:29:41.455417761 +0000 UTC m=+79.935216771 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eddc93f7-ea09-405e-9488-7c7e5f710ac2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tgp62" (UID: "eddc93f7-ea09-405e-9488-7c7e5f710ac2") : secret "insights-runtime-extractor-tls" not found Jul 06 11:29:33.455457 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:33.455453 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eddc93f7-ea09-405e-9488-7c7e5f710ac2-kube-rbac-proxy-cm podName:eddc93f7-ea09-405e-9488-7c7e5f710ac2 nodeName:}" failed. No retries permitted until 2026-07-06 11:29:41.455445098 +0000 UTC m=+79.935244116 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eddc93f7-ea09-405e-9488-7c7e5f710ac2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tgp62" (UID: "eddc93f7-ea09-405e-9488-7c7e5f710ac2") : configmap "kube-rbac-proxy" not found Jul 06 11:29:36.680515 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:36.680472 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e16731d-a528-49c9-8998-dcc1d0b44957-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-jjrll\" (UID: \"6e16731d-a528-49c9-8998-dcc1d0b44957\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jjrll" Jul 06 11:29:36.680944 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:36.680616 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 11:29:36.680944 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:36.680703 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e16731d-a528-49c9-8998-dcc1d0b44957-cluster-monitoring-operator-tls podName:6e16731d-a528-49c9-8998-dcc1d0b44957 nodeName:}" failed. No retries permitted until 2026-07-06 11:29:52.680669838 +0000 UTC m=+91.160468838 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6e16731d-a528-49c9-8998-dcc1d0b44957-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-jjrll" (UID: "6e16731d-a528-49c9-8998-dcc1d0b44957") : secret "cluster-monitoring-operator-tls" not found Jul 06 11:29:41.523891 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:41.523848 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eddc93f7-ea09-405e-9488-7c7e5f710ac2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tgp62\" (UID: \"eddc93f7-ea09-405e-9488-7c7e5f710ac2\") " pod="openshift-insights/insights-runtime-extractor-tgp62" Jul 06 11:29:41.524312 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:41.523901 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eddc93f7-ea09-405e-9488-7c7e5f710ac2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tgp62\" (UID: \"eddc93f7-ea09-405e-9488-7c7e5f710ac2\") " pod="openshift-insights/insights-runtime-extractor-tgp62" Jul 06 11:29:41.524312 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:41.523994 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 11:29:41.524312 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:29:41.524057 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eddc93f7-ea09-405e-9488-7c7e5f710ac2-insights-runtime-extractor-tls podName:eddc93f7-ea09-405e-9488-7c7e5f710ac2 nodeName:}" failed. No retries permitted until 2026-07-06 11:29:57.524038386 +0000 UTC m=+96.003837392 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eddc93f7-ea09-405e-9488-7c7e5f710ac2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tgp62" (UID: "eddc93f7-ea09-405e-9488-7c7e5f710ac2") : secret "insights-runtime-extractor-tls" not found Jul 06 11:29:41.524432 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:41.524360 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eddc93f7-ea09-405e-9488-7c7e5f710ac2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tgp62\" (UID: \"eddc93f7-ea09-405e-9488-7c7e5f710ac2\") " pod="openshift-insights/insights-runtime-extractor-tgp62" Jul 06 11:29:52.710105 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:52.709946 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e16731d-a528-49c9-8998-dcc1d0b44957-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-jjrll\" (UID: \"6e16731d-a528-49c9-8998-dcc1d0b44957\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jjrll" Jul 06 11:29:52.712516 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:52.712486 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6e16731d-a528-49c9-8998-dcc1d0b44957-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-jjrll\" (UID: \"6e16731d-a528-49c9-8998-dcc1d0b44957\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jjrll" Jul 06 11:29:52.977520 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:52.977441 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-w5cz8\"" Jul 06 11:29:52.985025 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:52.985002 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jjrll" Jul 06 11:29:53.125932 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:53.125896 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jjrll"] Jul 06 11:29:53.129187 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:29:53.129159 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e16731d_a528_49c9_8998_dcc1d0b44957.slice/crio-b9c61923c7c325054616a111d5b9f209da023b5ea5235a69f80ce2ae8ad160e1 WatchSource:0}: Error finding container b9c61923c7c325054616a111d5b9f209da023b5ea5235a69f80ce2ae8ad160e1: Status 404 returned error can't find the container with id b9c61923c7c325054616a111d5b9f209da023b5ea5235a69f80ce2ae8ad160e1 Jul 06 11:29:53.392977 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:53.392939 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jjrll" event={"ID":"6e16731d-a528-49c9-8998-dcc1d0b44957","Type":"ContainerStarted","Data":"b9c61923c7c325054616a111d5b9f209da023b5ea5235a69f80ce2ae8ad160e1"} Jul 06 11:29:55.401413 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:55.401369 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jjrll" event={"ID":"6e16731d-a528-49c9-8998-dcc1d0b44957","Type":"ContainerStarted","Data":"cb90c44007709156f90fac03d52adfc4c5d25ea9fa9ad495593c1446a4a2630d"} Jul 06 11:29:55.421626 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:55.421565 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jjrll" podStartSLOduration=33.664353046 podStartE2EDuration="35.421548661s" totalImagesPullingTime="1.757195615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:29:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:29:55.421031248 +0000 UTC m=+93.900830256" watchObservedRunningTime="2026-07-06 11:29:55.421548661 +0000 UTC m=+93.901347677" Jul 06 11:29:57.551772 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:57.551722 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eddc93f7-ea09-405e-9488-7c7e5f710ac2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tgp62\" (UID: \"eddc93f7-ea09-405e-9488-7c7e5f710ac2\") " pod="openshift-insights/insights-runtime-extractor-tgp62" Jul 06 11:29:57.554035 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:57.554012 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eddc93f7-ea09-405e-9488-7c7e5f710ac2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tgp62\" (UID: \"eddc93f7-ea09-405e-9488-7c7e5f710ac2\") " pod="openshift-insights/insights-runtime-extractor-tgp62" Jul 06 11:29:57.778051 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:57.778016 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tgp62" Jul 06 11:29:57.904441 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:57.904409 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tgp62"] Jul 06 11:29:57.907748 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:29:57.907718 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeddc93f7_ea09_405e_9488_7c7e5f710ac2.slice/crio-3363b77e43c2375f05b616a24f9647be708e078297fdfa1c4ddef77000a14a50 WatchSource:0}: Error finding container 3363b77e43c2375f05b616a24f9647be708e078297fdfa1c4ddef77000a14a50: Status 404 returned error can't find the container with id 3363b77e43c2375f05b616a24f9647be708e078297fdfa1c4ddef77000a14a50 Jul 06 11:29:58.412130 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:58.412078 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tgp62" event={"ID":"eddc93f7-ea09-405e-9488-7c7e5f710ac2","Type":"ContainerStarted","Data":"3363b77e43c2375f05b616a24f9647be708e078297fdfa1c4ddef77000a14a50"} Jul 06 11:29:58.655701 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:58.655474 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768f6965c6-x6ngz"] Jul 06 11:29:58.677956 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:58.677834 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768f6965c6-x6ngz" Jul 06 11:29:58.681895 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:58.681856 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768f6965c6-x6ngz"] Jul 06 11:29:58.682304 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:58.682279 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 06 11:29:58.683442 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:58.682518 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 06 11:29:58.683442 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:58.682779 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 06 11:29:58.683442 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:58.682985 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 06 11:29:58.683442 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:58.683221 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 06 11:29:58.683730 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:58.683448 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 06 11:29:58.685753 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:58.685732 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 06 11:29:58.762136 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:58.762092 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/25792da3-bd6a-4586-973e-53c797b22176-additional-service-ca\") pod \"cluster-proxy-proxy-agent-768f6965c6-x6ngz\" (UID: \"25792da3-bd6a-4586-973e-53c797b22176\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768f6965c6-x6ngz" Jul 06 11:29:58.762136 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:58.762135 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/25792da3-bd6a-4586-973e-53c797b22176-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-768f6965c6-x6ngz\" (UID: \"25792da3-bd6a-4586-973e-53c797b22176\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768f6965c6-x6ngz" Jul 06 11:29:58.762401 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:58.762171 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrkmm\" (UniqueName: \"kubernetes.io/projected/25792da3-bd6a-4586-973e-53c797b22176-kube-api-access-vrkmm\") pod \"cluster-proxy-proxy-agent-768f6965c6-x6ngz\" (UID: \"25792da3-bd6a-4586-973e-53c797b22176\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768f6965c6-x6ngz" Jul 06 11:29:58.762401 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:58.762205 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/25792da3-bd6a-4586-973e-53c797b22176-ca\") pod \"cluster-proxy-proxy-agent-768f6965c6-x6ngz\" (UID: \"25792da3-bd6a-4586-973e-53c797b22176\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768f6965c6-x6ngz" Jul 06 11:29:58.762401 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:58.762253 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/25792da3-bd6a-4586-973e-53c797b22176-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-768f6965c6-x6ngz\" (UID: \"25792da3-bd6a-4586-973e-53c797b22176\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768f6965c6-x6ngz" Jul 06 11:29:58.762401 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:58.762292 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/25792da3-bd6a-4586-973e-53c797b22176-hub\") pod \"cluster-proxy-proxy-agent-768f6965c6-x6ngz\" (UID: \"25792da3-bd6a-4586-973e-53c797b22176\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768f6965c6-x6ngz" Jul 06 11:29:58.862838 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:58.862791 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/25792da3-bd6a-4586-973e-53c797b22176-hub\") pod \"cluster-proxy-proxy-agent-768f6965c6-x6ngz\" (UID: \"25792da3-bd6a-4586-973e-53c797b22176\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768f6965c6-x6ngz" Jul 06 11:29:58.863144 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:58.862976 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/25792da3-bd6a-4586-973e-53c797b22176-additional-service-ca\") pod \"cluster-proxy-proxy-agent-768f6965c6-x6ngz\" (UID: \"25792da3-bd6a-4586-973e-53c797b22176\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768f6965c6-x6ngz" Jul 06 11:29:58.863144 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:58.863008 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/25792da3-bd6a-4586-973e-53c797b22176-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-768f6965c6-x6ngz\" (UID: \"25792da3-bd6a-4586-973e-53c797b22176\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768f6965c6-x6ngz" Jul 06 11:29:58.863144 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:58.863053 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrkmm\" (UniqueName: \"kubernetes.io/projected/25792da3-bd6a-4586-973e-53c797b22176-kube-api-access-vrkmm\") pod \"cluster-proxy-proxy-agent-768f6965c6-x6ngz\" (UID: \"25792da3-bd6a-4586-973e-53c797b22176\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768f6965c6-x6ngz" Jul 06 11:29:58.863144 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:58.863094 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/25792da3-bd6a-4586-973e-53c797b22176-ca\") pod \"cluster-proxy-proxy-agent-768f6965c6-x6ngz\" (UID: \"25792da3-bd6a-4586-973e-53c797b22176\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768f6965c6-x6ngz" Jul 06 11:29:58.863355 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:58.863161 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/25792da3-bd6a-4586-973e-53c797b22176-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-768f6965c6-x6ngz\" (UID: \"25792da3-bd6a-4586-973e-53c797b22176\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768f6965c6-x6ngz" Jul 06 11:29:58.864550 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:58.864522 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/25792da3-bd6a-4586-973e-53c797b22176-additional-service-ca\") pod \"cluster-proxy-proxy-agent-768f6965c6-x6ngz\" (UID: \"25792da3-bd6a-4586-973e-53c797b22176\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768f6965c6-x6ngz" Jul 06 11:29:58.866889 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:58.866862 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/25792da3-bd6a-4586-973e-53c797b22176-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-768f6965c6-x6ngz\" (UID: \"25792da3-bd6a-4586-973e-53c797b22176\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768f6965c6-x6ngz" Jul 06 11:29:58.867010 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:58.866951 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/25792da3-bd6a-4586-973e-53c797b22176-ca\") pod \"cluster-proxy-proxy-agent-768f6965c6-x6ngz\" (UID: \"25792da3-bd6a-4586-973e-53c797b22176\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768f6965c6-x6ngz" Jul 06 11:29:58.868342 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:58.868298 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/25792da3-bd6a-4586-973e-53c797b22176-hub\") pod \"cluster-proxy-proxy-agent-768f6965c6-x6ngz\" (UID: \"25792da3-bd6a-4586-973e-53c797b22176\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768f6965c6-x6ngz" Jul 06 11:29:58.869795 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:58.869771 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/25792da3-bd6a-4586-973e-53c797b22176-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-768f6965c6-x6ngz\" (UID: \"25792da3-bd6a-4586-973e-53c797b22176\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768f6965c6-x6ngz" Jul 06 11:29:58.879641 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:58.879573 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrkmm\" (UniqueName: \"kubernetes.io/projected/25792da3-bd6a-4586-973e-53c797b22176-kube-api-access-vrkmm\") pod \"cluster-proxy-proxy-agent-768f6965c6-x6ngz\" (UID: \"25792da3-bd6a-4586-973e-53c797b22176\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768f6965c6-x6ngz" Jul 06 11:29:59.003986 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:59.003890 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768f6965c6-x6ngz" Jul 06 11:29:59.175041 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:59.174897 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768f6965c6-x6ngz"] Jul 06 11:29:59.178109 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:29:59.178076 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25792da3_bd6a_4586_973e_53c797b22176.slice/crio-8d60ebbdd5cf5c7b7832d10ae8f22ca571027c86499b7f034c3bcb4fcf7a1e4c WatchSource:0}: Error finding container 8d60ebbdd5cf5c7b7832d10ae8f22ca571027c86499b7f034c3bcb4fcf7a1e4c: Status 404 returned error can't find the container with id 8d60ebbdd5cf5c7b7832d10ae8f22ca571027c86499b7f034c3bcb4fcf7a1e4c Jul 06 11:29:59.417276 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:29:59.417234 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768f6965c6-x6ngz" event={"ID":"25792da3-bd6a-4586-973e-53c797b22176","Type":"ContainerStarted","Data":"8d60ebbdd5cf5c7b7832d10ae8f22ca571027c86499b7f034c3bcb4fcf7a1e4c"} Jul 06 11:30:01.284640 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:01.284600 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-tls\") pod \"image-registry-577cf8786b-5q4rz\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:30:01.288310 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:01.288273 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-tls\") pod \"image-registry-577cf8786b-5q4rz\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:30:01.330211 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:01.330177 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rbs4n\"" Jul 06 11:30:01.337950 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:01.337920 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:30:01.487027 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:01.486988 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03ee7031-c88a-47a2-b37c-0349a1787567-metrics-tls\") pod \"dns-default-dvbz4\" (UID: \"03ee7031-c88a-47a2-b37c-0349a1787567\") " pod="openshift-dns/dns-default-dvbz4" Jul 06 11:30:01.489835 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:01.489810 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/03ee7031-c88a-47a2-b37c-0349a1787567-metrics-tls\") pod \"dns-default-dvbz4\" (UID: \"03ee7031-c88a-47a2-b37c-0349a1787567\") " pod="openshift-dns/dns-default-dvbz4" Jul 06 11:30:01.679297 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:01.679186 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bpbrx\"" Jul 06 11:30:01.686634 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:01.686605 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dvbz4" Jul 06 11:30:02.021186 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:02.020915 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dvbz4"] Jul 06 11:30:02.028653 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:30:02.027773 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ee7031_c88a_47a2_b37c_0349a1787567.slice/crio-b7f81827efd1e127db9086734980b949c4b210edffb751a081e6bbe9aa9ef76c WatchSource:0}: Error finding container b7f81827efd1e127db9086734980b949c4b210edffb751a081e6bbe9aa9ef76c: Status 404 returned error can't find the container with id b7f81827efd1e127db9086734980b949c4b210edffb751a081e6bbe9aa9ef76c Jul 06 11:30:02.052536 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:02.052470 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-577cf8786b-5q4rz"] Jul 06 11:30:02.054856 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:30:02.054824 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd435fc8e_6b2f_4f9c_94c8_be96d31f65a5.slice/crio-871953063b782bb9ea9815425b487dbfe12c3fa5c9527ebe2e487926efb52a57 WatchSource:0}: Error finding container 871953063b782bb9ea9815425b487dbfe12c3fa5c9527ebe2e487926efb52a57: Status 404 returned error can't find the container with id 871953063b782bb9ea9815425b487dbfe12c3fa5c9527ebe2e487926efb52a57 Jul 06 11:30:02.431723 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:02.431657 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dvbz4" event={"ID":"03ee7031-c88a-47a2-b37c-0349a1787567","Type":"ContainerStarted","Data":"b7f81827efd1e127db9086734980b949c4b210edffb751a081e6bbe9aa9ef76c"} Jul 06 11:30:02.433894 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:02.433856 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" event={"ID":"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5","Type":"ContainerStarted","Data":"91cd1721c2c75b5c064da8bac0497442a810074ebfb4a4c36e686be37f636913"} Jul 06 11:30:02.434059 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:02.433918 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" event={"ID":"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5","Type":"ContainerStarted","Data":"871953063b782bb9ea9815425b487dbfe12c3fa5c9527ebe2e487926efb52a57"} Jul 06 11:30:02.434194 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:02.434156 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:30:02.436228 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:02.436189 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tgp62" event={"ID":"eddc93f7-ea09-405e-9488-7c7e5f710ac2","Type":"ContainerStarted","Data":"574a37926529c8ae6569f6f44db722391f3995993c166dab2e895b02972b50f8"} Jul 06 11:30:02.455584 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:02.455514 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" podStartSLOduration=99.455493362 podStartE2EDuration="1m39.455493362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:30:02.455026519 +0000 UTC m=+100.934825544" watchObservedRunningTime="2026-07-06 11:30:02.455493362 +0000 UTC m=+100.935292377" Jul 06 11:30:04.339386 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:04.339352 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5dsbt" Jul 06 11:30:05.448298 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:05.448255 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tgp62" event={"ID":"eddc93f7-ea09-405e-9488-7c7e5f710ac2","Type":"ContainerStarted","Data":"9e5de02fbe023cade978f9ef76a2171d84807d41b7ee702821a7a68181492e97"} Jul 06 11:30:05.450539 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:05.450498 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768f6965c6-x6ngz" event={"ID":"25792da3-bd6a-4586-973e-53c797b22176","Type":"ContainerStarted","Data":"08f3de5e44309848ac07e9e27d78e5a9b1f3ea8d1097ca13c87fba58491e3b2c"} Jul 06 11:30:05.450711 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:05.450546 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768f6965c6-x6ngz" event={"ID":"25792da3-bd6a-4586-973e-53c797b22176","Type":"ContainerStarted","Data":"0f729ee9f74ab617fbf5ca08d38fc7598ffaccf96be2e3e9d454e5b2acdc32b1"} Jul 06 11:30:05.450711 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:05.450564 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768f6965c6-x6ngz" event={"ID":"25792da3-bd6a-4586-973e-53c797b22176","Type":"ContainerStarted","Data":"91a06e13a6d3119559d5be92e87190629356d4ebb89c1f7885f1e4565605e7ba"} Jul 06 11:30:05.473165 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:05.473092 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768f6965c6-x6ngz" podStartSLOduration=2.176308896 podStartE2EDuration="7.473067824s" totalImagesPullingTime="5.296758928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:29:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:30:05.471448498 +0000 UTC m=+103.951247523" watchObservedRunningTime="2026-07-06 11:30:05.473067824 +0000 UTC m=+103.952866842" Jul 06 11:30:08.463989 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:08.463955 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tgp62" event={"ID":"eddc93f7-ea09-405e-9488-7c7e5f710ac2","Type":"ContainerStarted","Data":"1f98dcbe928cab2c12b26f4ac9d4c956c065db66dfcf9d68ae8f3ae970d78c05"} Jul 06 11:30:08.465712 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:08.465687 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dvbz4" event={"ID":"03ee7031-c88a-47a2-b37c-0349a1787567","Type":"ContainerStarted","Data":"a73867491a9660c917971487e36fa2d766fa20a6d0063edefd79c0e1f0aeb119"} Jul 06 11:30:08.465830 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:08.465732 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dvbz4" event={"ID":"03ee7031-c88a-47a2-b37c-0349a1787567","Type":"ContainerStarted","Data":"f3e5fad41fcd3cd221ab84254dad6b6857a84aa161195c1da2783e57f4ca9e70"} Jul 06 11:30:08.465830 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:08.465815 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dvbz4" Jul 06 11:30:08.486733 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:08.486668 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tgp62" podStartSLOduration=33.631173871 podStartE2EDuration="43.486651937s" totalImagesPullingTime="9.855478066s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 11:29:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:30:08.486341686 +0000 UTC m=+106.966140715" watchObservedRunningTime="2026-07-06 11:30:08.486651937 +0000 UTC m=+106.966451004" Jul 06 11:30:08.506578 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:08.506529 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dvbz4" podStartSLOduration=65.887156306 podStartE2EDuration="1m11.506514544s" totalImagesPullingTime="5.619358238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:28:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:30:08.505538218 +0000 UTC m=+106.985337236" watchObservedRunningTime="2026-07-06 11:30:08.506514544 +0000 UTC m=+106.986313560" Jul 06 11:30:11.939829 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:11.939788 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-n4nfk"] Jul 06 11:30:11.942920 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:11.942898 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n4nfk" Jul 06 11:30:11.946018 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:11.945996 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 11:30:11.946216 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:11.946202 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 11:30:11.946258 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:11.946242 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-85mzg\"" Jul 06 11:30:11.947237 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:11.947220 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 11:30:11.947317 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:11.947275 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 11:30:12.079318 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:12.079281 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3bcac4b6-e83b-48d0-aacd-76e54565ac0d-root\") pod \"node-exporter-n4nfk\" (UID: \"3bcac4b6-e83b-48d0-aacd-76e54565ac0d\") " pod="openshift-monitoring/node-exporter-n4nfk" Jul 06 11:30:12.079495 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:12.079328 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3bcac4b6-e83b-48d0-aacd-76e54565ac0d-node-exporter-wtmp\") pod \"node-exporter-n4nfk\" (UID: \"3bcac4b6-e83b-48d0-aacd-76e54565ac0d\") " pod="openshift-monitoring/node-exporter-n4nfk" Jul 06 11:30:12.079495 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:12.079352 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3bcac4b6-e83b-48d0-aacd-76e54565ac0d-node-exporter-tls\") pod \"node-exporter-n4nfk\" (UID: \"3bcac4b6-e83b-48d0-aacd-76e54565ac0d\") " pod="openshift-monitoring/node-exporter-n4nfk" Jul 06 11:30:12.079495 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:12.079372 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bcac4b6-e83b-48d0-aacd-76e54565ac0d-sys\") pod \"node-exporter-n4nfk\" (UID: \"3bcac4b6-e83b-48d0-aacd-76e54565ac0d\") " pod="openshift-monitoring/node-exporter-n4nfk" Jul 06 11:30:12.079495 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:12.079392 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3bcac4b6-e83b-48d0-aacd-76e54565ac0d-node-exporter-accelerators-collector-config\") pod \"node-exporter-n4nfk\" (UID: \"3bcac4b6-e83b-48d0-aacd-76e54565ac0d\") " pod="openshift-monitoring/node-exporter-n4nfk" Jul 06 11:30:12.079495 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:12.079429 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g9fh\" (UniqueName: \"kubernetes.io/projected/3bcac4b6-e83b-48d0-aacd-76e54565ac0d-kube-api-access-5g9fh\") pod \"node-exporter-n4nfk\" (UID: \"3bcac4b6-e83b-48d0-aacd-76e54565ac0d\") " pod="openshift-monitoring/node-exporter-n4nfk" Jul 06 11:30:12.079495 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:12.079486 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3bcac4b6-e83b-48d0-aacd-76e54565ac0d-node-exporter-textfile\") pod \"node-exporter-n4nfk\" (UID: \"3bcac4b6-e83b-48d0-aacd-76e54565ac0d\") " pod="openshift-monitoring/node-exporter-n4nfk" Jul 06 11:30:12.079755 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:12.079515 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bcac4b6-e83b-48d0-aacd-76e54565ac0d-metrics-client-ca\") pod \"node-exporter-n4nfk\" (UID: \"3bcac4b6-e83b-48d0-aacd-76e54565ac0d\") " pod="openshift-monitoring/node-exporter-n4nfk" Jul 06 11:30:12.079755 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:12.079535 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3bcac4b6-e83b-48d0-aacd-76e54565ac0d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n4nfk\" (UID: \"3bcac4b6-e83b-48d0-aacd-76e54565ac0d\") " pod="openshift-monitoring/node-exporter-n4nfk" Jul 06 11:30:12.180164 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:12.180112 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3bcac4b6-e83b-48d0-aacd-76e54565ac0d-node-exporter-accelerators-collector-config\") pod \"node-exporter-n4nfk\" (UID: \"3bcac4b6-e83b-48d0-aacd-76e54565ac0d\") " pod="openshift-monitoring/node-exporter-n4nfk" Jul 06 11:30:12.180372 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:12.180190 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g9fh\" (UniqueName: \"kubernetes.io/projected/3bcac4b6-e83b-48d0-aacd-76e54565ac0d-kube-api-access-5g9fh\") pod \"node-exporter-n4nfk\" (UID: \"3bcac4b6-e83b-48d0-aacd-76e54565ac0d\") " pod="openshift-monitoring/node-exporter-n4nfk" Jul 06 11:30:12.180372 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:12.180216 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3bcac4b6-e83b-48d0-aacd-76e54565ac0d-node-exporter-textfile\") pod \"node-exporter-n4nfk\" (UID: \"3bcac4b6-e83b-48d0-aacd-76e54565ac0d\") " pod="openshift-monitoring/node-exporter-n4nfk" Jul 06 11:30:12.180372 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:12.180241 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bcac4b6-e83b-48d0-aacd-76e54565ac0d-metrics-client-ca\") pod \"node-exporter-n4nfk\" (UID: \"3bcac4b6-e83b-48d0-aacd-76e54565ac0d\") " pod="openshift-monitoring/node-exporter-n4nfk" Jul 06 11:30:12.180372 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:12.180262 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3bcac4b6-e83b-48d0-aacd-76e54565ac0d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n4nfk\" (UID: \"3bcac4b6-e83b-48d0-aacd-76e54565ac0d\") " pod="openshift-monitoring/node-exporter-n4nfk" Jul 06 11:30:12.180372 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:12.180319 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3bcac4b6-e83b-48d0-aacd-76e54565ac0d-root\") pod \"node-exporter-n4nfk\" (UID: \"3bcac4b6-e83b-48d0-aacd-76e54565ac0d\") " pod="openshift-monitoring/node-exporter-n4nfk" Jul 06 11:30:12.180626 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:12.180384 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3bcac4b6-e83b-48d0-aacd-76e54565ac0d-node-exporter-wtmp\") pod \"node-exporter-n4nfk\" (UID: \"3bcac4b6-e83b-48d0-aacd-76e54565ac0d\") " pod="openshift-monitoring/node-exporter-n4nfk" Jul 06 11:30:12.180626 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:12.180412 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3bcac4b6-e83b-48d0-aacd-76e54565ac0d-node-exporter-tls\") pod \"node-exporter-n4nfk\" (UID: \"3bcac4b6-e83b-48d0-aacd-76e54565ac0d\") " pod="openshift-monitoring/node-exporter-n4nfk" Jul 06 11:30:12.180626 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:12.180416 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3bcac4b6-e83b-48d0-aacd-76e54565ac0d-root\") pod \"node-exporter-n4nfk\" (UID: \"3bcac4b6-e83b-48d0-aacd-76e54565ac0d\") " pod="openshift-monitoring/node-exporter-n4nfk" Jul 06 11:30:12.180626 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:12.180435 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bcac4b6-e83b-48d0-aacd-76e54565ac0d-sys\") pod \"node-exporter-n4nfk\" (UID: \"3bcac4b6-e83b-48d0-aacd-76e54565ac0d\") " pod="openshift-monitoring/node-exporter-n4nfk" Jul 06 11:30:12.180626 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:12.180619 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3bcac4b6-e83b-48d0-aacd-76e54565ac0d-node-exporter-textfile\") pod \"node-exporter-n4nfk\" (UID: \"3bcac4b6-e83b-48d0-aacd-76e54565ac0d\") " pod="openshift-monitoring/node-exporter-n4nfk" Jul 06 11:30:12.180852 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:12.180619 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3bcac4b6-e83b-48d0-aacd-76e54565ac0d-node-exporter-wtmp\") pod \"node-exporter-n4nfk\" (UID: \"3bcac4b6-e83b-48d0-aacd-76e54565ac0d\") " pod="openshift-monitoring/node-exporter-n4nfk" Jul 06 11:30:12.180852 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:12.180652 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bcac4b6-e83b-48d0-aacd-76e54565ac0d-sys\") pod \"node-exporter-n4nfk\" (UID: \"3bcac4b6-e83b-48d0-aacd-76e54565ac0d\") " pod="openshift-monitoring/node-exporter-n4nfk" Jul 06 11:30:12.180927 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:12.180892 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3bcac4b6-e83b-48d0-aacd-76e54565ac0d-node-exporter-accelerators-collector-config\") pod \"node-exporter-n4nfk\" (UID: \"3bcac4b6-e83b-48d0-aacd-76e54565ac0d\") " pod="openshift-monitoring/node-exporter-n4nfk" Jul 06 11:30:12.180927 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:12.180892 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bcac4b6-e83b-48d0-aacd-76e54565ac0d-metrics-client-ca\") pod \"node-exporter-n4nfk\" (UID: \"3bcac4b6-e83b-48d0-aacd-76e54565ac0d\") " pod="openshift-monitoring/node-exporter-n4nfk" Jul 06 11:30:12.182860 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:12.182837 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3bcac4b6-e83b-48d0-aacd-76e54565ac0d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n4nfk\" (UID: \"3bcac4b6-e83b-48d0-aacd-76e54565ac0d\") " pod="openshift-monitoring/node-exporter-n4nfk" Jul 06 11:30:12.182907 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:12.182891 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3bcac4b6-e83b-48d0-aacd-76e54565ac0d-node-exporter-tls\") pod \"node-exporter-n4nfk\" (UID: \"3bcac4b6-e83b-48d0-aacd-76e54565ac0d\") " pod="openshift-monitoring/node-exporter-n4nfk" Jul 06 11:30:12.189252 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:12.189226 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g9fh\" (UniqueName: \"kubernetes.io/projected/3bcac4b6-e83b-48d0-aacd-76e54565ac0d-kube-api-access-5g9fh\") pod \"node-exporter-n4nfk\" (UID: \"3bcac4b6-e83b-48d0-aacd-76e54565ac0d\") " pod="openshift-monitoring/node-exporter-n4nfk" Jul 06 11:30:12.252378 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:12.252301 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n4nfk" Jul 06 11:30:12.263197 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:30:12.263168 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bcac4b6_e83b_48d0_aacd_76e54565ac0d.slice/crio-86850052d8c3ae2adbaf2d235a68254319a5f0004f939326a7efd41ba910c718 WatchSource:0}: Error finding container 86850052d8c3ae2adbaf2d235a68254319a5f0004f939326a7efd41ba910c718: Status 404 returned error can't find the container with id 86850052d8c3ae2adbaf2d235a68254319a5f0004f939326a7efd41ba910c718 Jul 06 11:30:12.478999 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:12.478961 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n4nfk" event={"ID":"3bcac4b6-e83b-48d0-aacd-76e54565ac0d","Type":"ContainerStarted","Data":"86850052d8c3ae2adbaf2d235a68254319a5f0004f939326a7efd41ba910c718"} Jul 06 11:30:14.489543 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:14.489505 2568 generic.go:356] "Generic (PLEG): container finished" podID="3bcac4b6-e83b-48d0-aacd-76e54565ac0d" containerID="a38b3b75cb791af8d35d9920e10b1abc4e9a51f93e685fb74ae045b5b7abbb58" exitCode=0 Jul 06 11:30:14.489975 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:14.489578 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n4nfk" event={"ID":"3bcac4b6-e83b-48d0-aacd-76e54565ac0d","Type":"ContainerDied","Data":"a38b3b75cb791af8d35d9920e10b1abc4e9a51f93e685fb74ae045b5b7abbb58"} Jul 06 11:30:15.495388 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:15.495348 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n4nfk" event={"ID":"3bcac4b6-e83b-48d0-aacd-76e54565ac0d","Type":"ContainerStarted","Data":"d6d76c9430bb9be5b9c570b63739ca05744666b292a2222ed87be97ef2c68cb4"} Jul 06 11:30:15.495871 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:15.495403 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n4nfk" event={"ID":"3bcac4b6-e83b-48d0-aacd-76e54565ac0d","Type":"ContainerStarted","Data":"ac376b23e462219b0b30872575ca989f5dae4cbb8fdc2b894e107508eb630f46"} Jul 06 11:30:15.521173 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:15.521117 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-n4nfk" podStartSLOduration=3.090673771 podStartE2EDuration="4.521099722s" totalImagesPullingTime="1.365251434s" totalInitContainerRuntime="65.174517ms" isStatefulPod=true podCreationTimestamp="2026-07-06 11:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:30:15.51904185 +0000 UTC m=+113.998840867" watchObservedRunningTime="2026-07-06 11:30:15.521099722 +0000 UTC m=+114.000898805" Jul 06 11:30:18.473138 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:18.473102 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dvbz4" Jul 06 11:30:21.259834 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:21.259799 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-577cf8786b-5q4rz"] Jul 06 11:30:21.264096 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:21.264072 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:30:31.947809 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:31.947764 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6730cd7d-c083-452b-b123-e2e9f92745a1-metrics-certs\") pod \"network-metrics-daemon-rgmk8\" (UID: \"6730cd7d-c083-452b-b123-e2e9f92745a1\") " pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:30:31.950121 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:31.950097 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6730cd7d-c083-452b-b123-e2e9f92745a1-metrics-certs\") pod \"network-metrics-daemon-rgmk8\" (UID: \"6730cd7d-c083-452b-b123-e2e9f92745a1\") " pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:30:32.074857 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:32.074829 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w64mh\"" Jul 06 11:30:32.078462 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:32.078445 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rgmk8" Jul 06 11:30:32.235002 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:32.234915 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rgmk8"] Jul 06 11:30:32.238146 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:30:32.238109 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6730cd7d_c083_452b_b123_e2e9f92745a1.slice/crio-0e1df14f84808ab41285b8909672f583a9c47974d53a62922e29c3944aab9f73 WatchSource:0}: Error finding container 0e1df14f84808ab41285b8909672f583a9c47974d53a62922e29c3944aab9f73: Status 404 returned error can't find the container with id 0e1df14f84808ab41285b8909672f583a9c47974d53a62922e29c3944aab9f73 Jul 06 11:30:32.554629 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:32.554543 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rgmk8" event={"ID":"6730cd7d-c083-452b-b123-e2e9f92745a1","Type":"ContainerStarted","Data":"0e1df14f84808ab41285b8909672f583a9c47974d53a62922e29c3944aab9f73"} Jul 06 11:30:33.560143 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:33.560100 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rgmk8" event={"ID":"6730cd7d-c083-452b-b123-e2e9f92745a1","Type":"ContainerStarted","Data":"b811a78d1205feb03752ab47c11766ec55ae0e16e50192b09f4717f9c4338562"} Jul 06 11:30:34.565167 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:34.565129 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rgmk8" event={"ID":"6730cd7d-c083-452b-b123-e2e9f92745a1","Type":"ContainerStarted","Data":"c2d3c416355a81fc2316c221f5cf75b4efc06853e92a4b1a923fdfd5df9126d0"} Jul 06 11:30:34.586513 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:34.586454 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rgmk8" podStartSLOduration=130.419143266 podStartE2EDuration="2m11.586433849s" totalImagesPullingTime="1.167290583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:28:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:30:34.585587311 +0000 UTC m=+133.065386326" watchObservedRunningTime="2026-07-06 11:30:34.586433849 +0000 UTC m=+133.066232866" Jul 06 11:30:45.605100 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:45.605003 2568 generic.go:356] "Generic (PLEG): container finished" podID="2864a14b-0f18-45e1-8c00-129a0c26570c" containerID="ca4d22619fbe9f388b9b1a3e5fe8b1ba63f610511183ca9cc78535519a3b67b9" exitCode=0 Jul 06 11:30:45.605100 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:45.605085 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5cq2k" event={"ID":"2864a14b-0f18-45e1-8c00-129a0c26570c","Type":"ContainerDied","Data":"ca4d22619fbe9f388b9b1a3e5fe8b1ba63f610511183ca9cc78535519a3b67b9"} Jul 06 11:30:45.605544 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:45.605400 2568 scope.go:117] "RemoveContainer" containerID="ca4d22619fbe9f388b9b1a3e5fe8b1ba63f610511183ca9cc78535519a3b67b9" Jul 06 11:30:46.279999 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.279946 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" podUID="d435fc8e-6b2f-4f9c-94c8-be96d31f65a5" containerName="registry" containerID="cri-o://91cd1721c2c75b5c064da8bac0497442a810074ebfb4a4c36e686be37f636913" gracePeriod=30 Jul 06 11:30:46.524126 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.524103 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:30:46.539666 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.539601 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-installation-pull-secrets\") pod \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " Jul 06 11:30:46.539793 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.539773 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-ca-trust-extracted\") pod \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " Jul 06 11:30:46.539833 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.539817 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-tls\") pod \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " Jul 06 11:30:46.539867 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.539858 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-trusted-ca\") pod \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " Jul 06 11:30:46.539934 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.539919 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-certificates\") pod \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " Jul 06 11:30:46.539985 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.539954 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-image-registry-private-configuration\") pod \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " Jul 06 11:30:46.540051 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.539982 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-bound-sa-token\") pod \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " Jul 06 11:30:46.540429 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.540160 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kq6fr\" (UniqueName: \"kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-kube-api-access-kq6fr\") pod \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\" (UID: \"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5\") " Jul 06 11:30:46.540429 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.540403 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d435fc8e-6b2f-4f9c-94c8-be96d31f65a5" (UID: "d435fc8e-6b2f-4f9c-94c8-be96d31f65a5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 11:30:46.540710 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.540662 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d435fc8e-6b2f-4f9c-94c8-be96d31f65a5" (UID: "d435fc8e-6b2f-4f9c-94c8-be96d31f65a5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 11:30:46.542954 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.542901 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d435fc8e-6b2f-4f9c-94c8-be96d31f65a5" (UID: "d435fc8e-6b2f-4f9c-94c8-be96d31f65a5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 11:30:46.543159 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.543129 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d435fc8e-6b2f-4f9c-94c8-be96d31f65a5" (UID: "d435fc8e-6b2f-4f9c-94c8-be96d31f65a5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 11:30:46.543277 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.543220 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d435fc8e-6b2f-4f9c-94c8-be96d31f65a5" (UID: "d435fc8e-6b2f-4f9c-94c8-be96d31f65a5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 11:30:46.543368 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.543350 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-kube-api-access-kq6fr" (OuterVolumeSpecName: "kube-api-access-kq6fr") pod "d435fc8e-6b2f-4f9c-94c8-be96d31f65a5" (UID: "d435fc8e-6b2f-4f9c-94c8-be96d31f65a5"). InnerVolumeSpecName "kube-api-access-kq6fr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 11:30:46.543554 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.543535 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d435fc8e-6b2f-4f9c-94c8-be96d31f65a5" (UID: "d435fc8e-6b2f-4f9c-94c8-be96d31f65a5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 11:30:46.550195 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.550166 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d435fc8e-6b2f-4f9c-94c8-be96d31f65a5" (UID: "d435fc8e-6b2f-4f9c-94c8-be96d31f65a5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:30:46.609847 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.609806 2568 generic.go:356] "Generic (PLEG): container finished" podID="d435fc8e-6b2f-4f9c-94c8-be96d31f65a5" containerID="91cd1721c2c75b5c064da8bac0497442a810074ebfb4a4c36e686be37f636913" exitCode=0 Jul 06 11:30:46.610303 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.609866 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" Jul 06 11:30:46.610303 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.609892 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" event={"ID":"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5","Type":"ContainerDied","Data":"91cd1721c2c75b5c064da8bac0497442a810074ebfb4a4c36e686be37f636913"} Jul 06 11:30:46.610303 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.609944 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-577cf8786b-5q4rz" event={"ID":"d435fc8e-6b2f-4f9c-94c8-be96d31f65a5","Type":"ContainerDied","Data":"871953063b782bb9ea9815425b487dbfe12c3fa5c9527ebe2e487926efb52a57"} Jul 06 11:30:46.610303 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.609968 2568 scope.go:117] "RemoveContainer" containerID="91cd1721c2c75b5c064da8bac0497442a810074ebfb4a4c36e686be37f636913" Jul 06 11:30:46.612132 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.612108 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-5cq2k" event={"ID":"2864a14b-0f18-45e1-8c00-129a0c26570c","Type":"ContainerStarted","Data":"6366ab40ccbe93ef7966df4084827d13e665f157c58c97056778bf7d32c30105"} Jul 06 11:30:46.618938 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.618914 2568 scope.go:117] "RemoveContainer" containerID="91cd1721c2c75b5c064da8bac0497442a810074ebfb4a4c36e686be37f636913" Jul 06 11:30:46.619241 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:30:46.619212 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91cd1721c2c75b5c064da8bac0497442a810074ebfb4a4c36e686be37f636913\": container with ID starting with 91cd1721c2c75b5c064da8bac0497442a810074ebfb4a4c36e686be37f636913 not found: ID does not exist" containerID="91cd1721c2c75b5c064da8bac0497442a810074ebfb4a4c36e686be37f636913" Jul 06 11:30:46.619304 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.619256 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91cd1721c2c75b5c064da8bac0497442a810074ebfb4a4c36e686be37f636913"} err="failed to get container status \"91cd1721c2c75b5c064da8bac0497442a810074ebfb4a4c36e686be37f636913\": rpc error: code = NotFound desc = could not find container \"91cd1721c2c75b5c064da8bac0497442a810074ebfb4a4c36e686be37f636913\": container with ID starting with 91cd1721c2c75b5c064da8bac0497442a810074ebfb4a4c36e686be37f636913 not found: ID does not exist" Jul 06 11:30:46.641231 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.641204 2568 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-certificates\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:30:46.641231 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.641230 2568 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-image-registry-private-configuration\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:30:46.641449 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.641247 2568 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-bound-sa-token\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:30:46.641449 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.641258 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq6fr\" (UniqueName: \"kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-kube-api-access-kq6fr\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:30:46.641449 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.641267 2568 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-installation-pull-secrets\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:30:46.641449 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.641275 2568 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-ca-trust-extracted\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:30:46.641449 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.641283 2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-registry-tls\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:30:46.641449 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.641294 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5-trusted-ca\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:30:46.648903 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.648869 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-577cf8786b-5q4rz"] Jul 06 11:30:46.660258 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:46.660218 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-577cf8786b-5q4rz"] Jul 06 11:30:48.050759 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:48.050713 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d435fc8e-6b2f-4f9c-94c8-be96d31f65a5" path="/var/lib/kubelet/pods/d435fc8e-6b2f-4f9c-94c8-be96d31f65a5/volumes" Jul 06 11:30:50.629490 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:50.629394 2568 generic.go:356] "Generic (PLEG): container finished" podID="eba80240-4cdd-4537-a223-616b0196c115" containerID="d3c49e80605d644c6b456d9417a8aaea4ddb4b468fc2ed358e84838182268285" exitCode=0 Jul 06 11:30:50.629490 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:50.629471 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6knbf" event={"ID":"eba80240-4cdd-4537-a223-616b0196c115","Type":"ContainerDied","Data":"d3c49e80605d644c6b456d9417a8aaea4ddb4b468fc2ed358e84838182268285"} Jul 06 11:30:50.629996 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:50.629839 2568 scope.go:117] "RemoveContainer" containerID="d3c49e80605d644c6b456d9417a8aaea4ddb4b468fc2ed358e84838182268285" Jul 06 11:30:51.634804 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:30:51.634770 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-6knbf" event={"ID":"eba80240-4cdd-4537-a223-616b0196c115","Type":"ContainerStarted","Data":"26b7150cf096f11e02743fe18d34b848adc92ba8462674e5bbb85617c1b20fe6"} Jul 06 11:31:00.665331 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:31:00.665295 2568 generic.go:356] "Generic (PLEG): container finished" podID="8a12983e-eac8-4b8b-8baf-688b9f7c4907" containerID="f1a9463088605d9bf10c266789a4a2763978d18100d2e18cecf84f4ef9f2d27e" exitCode=0 Jul 06 11:31:00.665817 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:31:00.665369 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-gsnkd" event={"ID":"8a12983e-eac8-4b8b-8baf-688b9f7c4907","Type":"ContainerDied","Data":"f1a9463088605d9bf10c266789a4a2763978d18100d2e18cecf84f4ef9f2d27e"} Jul 06 11:31:00.665817 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:31:00.665725 2568 scope.go:117] "RemoveContainer" containerID="f1a9463088605d9bf10c266789a4a2763978d18100d2e18cecf84f4ef9f2d27e" Jul 06 11:31:01.671272 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:31:01.671241 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-gsnkd" event={"ID":"8a12983e-eac8-4b8b-8baf-688b9f7c4907","Type":"ContainerStarted","Data":"92c8e3886eecf524d338a18d9950a3c2e74e241fc18f6890eb3699c4e7804e1a"} Jul 06 11:33:21.956850 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:33:21.956818 2568 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 11:34:01.409527 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:01.409491 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6gwq"] Jul 06 11:34:01.411826 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:01.409742 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d435fc8e-6b2f-4f9c-94c8-be96d31f65a5" containerName="registry" Jul 06 11:34:01.411826 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:01.409753 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d435fc8e-6b2f-4f9c-94c8-be96d31f65a5" containerName="registry" Jul 06 11:34:01.411826 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:01.409793 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d435fc8e-6b2f-4f9c-94c8-be96d31f65a5" containerName="registry" Jul 06 11:34:01.412699 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:01.412665 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6gwq" Jul 06 11:34:01.415813 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:01.415791 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 11:34:01.415968 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:01.415913 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 11:34:01.417120 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:01.417102 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xkzgz\"" Jul 06 11:34:01.422040 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:01.422018 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6gwq"] Jul 06 11:34:01.481258 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:01.481223 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsvll\" (UniqueName: \"kubernetes.io/projected/59e8d538-95b5-411e-bd53-81778d3c444c-kube-api-access-hsvll\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6gwq\" (UID: \"59e8d538-95b5-411e-bd53-81778d3c444c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6gwq" Jul 06 11:34:01.481471 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:01.481321 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59e8d538-95b5-411e-bd53-81778d3c444c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6gwq\" (UID: \"59e8d538-95b5-411e-bd53-81778d3c444c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6gwq" Jul 06 11:34:01.481471 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:01.481347 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59e8d538-95b5-411e-bd53-81778d3c444c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6gwq\" (UID: \"59e8d538-95b5-411e-bd53-81778d3c444c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6gwq" Jul 06 11:34:01.582065 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:01.582026 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59e8d538-95b5-411e-bd53-81778d3c444c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6gwq\" (UID: \"59e8d538-95b5-411e-bd53-81778d3c444c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6gwq" Jul 06 11:34:01.582065 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:01.582065 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59e8d538-95b5-411e-bd53-81778d3c444c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6gwq\" (UID: \"59e8d538-95b5-411e-bd53-81778d3c444c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6gwq" Jul 06 11:34:01.582306 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:01.582090 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsvll\" (UniqueName: \"kubernetes.io/projected/59e8d538-95b5-411e-bd53-81778d3c444c-kube-api-access-hsvll\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6gwq\" (UID: \"59e8d538-95b5-411e-bd53-81778d3c444c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6gwq" Jul 06 11:34:01.582510 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:01.582488 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59e8d538-95b5-411e-bd53-81778d3c444c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6gwq\" (UID: \"59e8d538-95b5-411e-bd53-81778d3c444c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6gwq" Jul 06 11:34:01.582543 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:01.582493 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59e8d538-95b5-411e-bd53-81778d3c444c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6gwq\" (UID: \"59e8d538-95b5-411e-bd53-81778d3c444c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6gwq" Jul 06 11:34:01.591033 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:01.591002 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsvll\" (UniqueName: \"kubernetes.io/projected/59e8d538-95b5-411e-bd53-81778d3c444c-kube-api-access-hsvll\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6gwq\" (UID: \"59e8d538-95b5-411e-bd53-81778d3c444c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6gwq" Jul 06 11:34:01.721663 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:01.721572 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6gwq" Jul 06 11:34:01.849901 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:01.849868 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6gwq"] Jul 06 11:34:01.852750 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:34:01.852721 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59e8d538_95b5_411e_bd53_81778d3c444c.slice/crio-e0707ff25de208de6845d047fc81f2f651ffbd84cf5d768f8a1f814b7c1d5f78 WatchSource:0}: Error finding container e0707ff25de208de6845d047fc81f2f651ffbd84cf5d768f8a1f814b7c1d5f78: Status 404 returned error can't find the container with id e0707ff25de208de6845d047fc81f2f651ffbd84cf5d768f8a1f814b7c1d5f78 Jul 06 11:34:01.854738 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:01.854719 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 11:34:02.275404 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:02.275370 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6gwq" event={"ID":"59e8d538-95b5-411e-bd53-81778d3c444c","Type":"ContainerStarted","Data":"e0707ff25de208de6845d047fc81f2f651ffbd84cf5d768f8a1f814b7c1d5f78"} Jul 06 11:34:08.297894 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:08.297859 2568 generic.go:356] "Generic (PLEG): container finished" podID="59e8d538-95b5-411e-bd53-81778d3c444c" containerID="9765dbba1634e6c409e54cff087729024ff1759085f7f34c269f373b61194ddd" exitCode=0 Jul 06 11:34:08.298277 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:08.297903 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6gwq" event={"ID":"59e8d538-95b5-411e-bd53-81778d3c444c","Type":"ContainerDied","Data":"9765dbba1634e6c409e54cff087729024ff1759085f7f34c269f373b61194ddd"} Jul 06 11:34:11.311496 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:11.311454 2568 generic.go:356] "Generic (PLEG): container finished" podID="59e8d538-95b5-411e-bd53-81778d3c444c" containerID="1bb388780ddc30b8bd04f681ebaf368e59357c89139099208fa60355e6888028" exitCode=0 Jul 06 11:34:11.311900 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:11.311510 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6gwq" event={"ID":"59e8d538-95b5-411e-bd53-81778d3c444c","Type":"ContainerDied","Data":"1bb388780ddc30b8bd04f681ebaf368e59357c89139099208fa60355e6888028"} Jul 06 11:34:19.342063 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:19.342024 2568 generic.go:356] "Generic (PLEG): container finished" podID="59e8d538-95b5-411e-bd53-81778d3c444c" containerID="2bb755b24473ee3ddf5a9bc057f7ee1577447b37869623ccd709c07bd84119ea" exitCode=0 Jul 06 11:34:19.342063 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:19.342066 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6gwq" event={"ID":"59e8d538-95b5-411e-bd53-81778d3c444c","Type":"ContainerDied","Data":"2bb755b24473ee3ddf5a9bc057f7ee1577447b37869623ccd709c07bd84119ea"} Jul 06 11:34:20.468862 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:20.468831 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6gwq" Jul 06 11:34:20.640143 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:20.640113 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsvll\" (UniqueName: \"kubernetes.io/projected/59e8d538-95b5-411e-bd53-81778d3c444c-kube-api-access-hsvll\") pod \"59e8d538-95b5-411e-bd53-81778d3c444c\" (UID: \"59e8d538-95b5-411e-bd53-81778d3c444c\") " Jul 06 11:34:20.640301 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:20.640169 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59e8d538-95b5-411e-bd53-81778d3c444c-util\") pod \"59e8d538-95b5-411e-bd53-81778d3c444c\" (UID: \"59e8d538-95b5-411e-bd53-81778d3c444c\") " Jul 06 11:34:20.640301 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:20.640196 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59e8d538-95b5-411e-bd53-81778d3c444c-bundle\") pod \"59e8d538-95b5-411e-bd53-81778d3c444c\" (UID: \"59e8d538-95b5-411e-bd53-81778d3c444c\") " Jul 06 11:34:20.641035 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:20.641005 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59e8d538-95b5-411e-bd53-81778d3c444c-bundle" (OuterVolumeSpecName: "bundle") pod "59e8d538-95b5-411e-bd53-81778d3c444c" (UID: "59e8d538-95b5-411e-bd53-81778d3c444c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:34:20.642441 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:20.642420 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59e8d538-95b5-411e-bd53-81778d3c444c-kube-api-access-hsvll" (OuterVolumeSpecName: "kube-api-access-hsvll") pod "59e8d538-95b5-411e-bd53-81778d3c444c" (UID: "59e8d538-95b5-411e-bd53-81778d3c444c"). InnerVolumeSpecName "kube-api-access-hsvll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 11:34:20.644808 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:20.644786 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59e8d538-95b5-411e-bd53-81778d3c444c-util" (OuterVolumeSpecName: "util") pod "59e8d538-95b5-411e-bd53-81778d3c444c" (UID: "59e8d538-95b5-411e-bd53-81778d3c444c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:34:20.741278 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:20.741242 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsvll\" (UniqueName: \"kubernetes.io/projected/59e8d538-95b5-411e-bd53-81778d3c444c-kube-api-access-hsvll\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:34:20.741278 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:20.741273 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59e8d538-95b5-411e-bd53-81778d3c444c-util\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:34:20.741278 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:20.741283 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59e8d538-95b5-411e-bd53-81778d3c444c-bundle\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:34:21.351521 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:21.351494 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6gwq" Jul 06 11:34:21.351521 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:21.351502 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q6gwq" event={"ID":"59e8d538-95b5-411e-bd53-81778d3c444c","Type":"ContainerDied","Data":"e0707ff25de208de6845d047fc81f2f651ffbd84cf5d768f8a1f814b7c1d5f78"} Jul 06 11:34:21.351763 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:21.351547 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0707ff25de208de6845d047fc81f2f651ffbd84cf5d768f8a1f814b7c1d5f78" Jul 06 11:34:24.590530 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:24.590489 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wm6n6"] Jul 06 11:34:24.590957 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:24.590755 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59e8d538-95b5-411e-bd53-81778d3c444c" containerName="pull" Jul 06 11:34:24.590957 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:24.590768 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="59e8d538-95b5-411e-bd53-81778d3c444c" containerName="pull" Jul 06 11:34:24.590957 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:24.590784 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59e8d538-95b5-411e-bd53-81778d3c444c" containerName="extract" Jul 06 11:34:24.590957 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:24.590789 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="59e8d538-95b5-411e-bd53-81778d3c444c" containerName="extract" Jul 06 11:34:24.590957 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:24.590796 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59e8d538-95b5-411e-bd53-81778d3c444c" containerName="util" Jul 06 11:34:24.590957 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:24.590801 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="59e8d538-95b5-411e-bd53-81778d3c444c" containerName="util" Jul 06 11:34:24.590957 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:24.590852 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="59e8d538-95b5-411e-bd53-81778d3c444c" containerName="extract" Jul 06 11:34:24.594361 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:24.594346 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wm6n6" Jul 06 11:34:24.597250 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:24.597225 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 06 11:34:24.597439 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:24.597418 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-9qf6f\"" Jul 06 11:34:24.597526 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:24.597465 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 06 11:34:24.616000 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:24.615972 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wm6n6"] Jul 06 11:34:24.664055 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:24.664016 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/874a9773-1fa4-4640-b50f-574a6e958ce5-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wm6n6\" (UID: \"874a9773-1fa4-4640-b50f-574a6e958ce5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wm6n6" Jul 06 11:34:24.664230 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:24.664061 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkcns\" (UniqueName: \"kubernetes.io/projected/874a9773-1fa4-4640-b50f-574a6e958ce5-kube-api-access-tkcns\") pod \"cert-manager-operator-controller-manager-55c444bf86-wm6n6\" (UID: \"874a9773-1fa4-4640-b50f-574a6e958ce5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wm6n6" Jul 06 11:34:24.765091 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:24.765061 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkcns\" (UniqueName: \"kubernetes.io/projected/874a9773-1fa4-4640-b50f-574a6e958ce5-kube-api-access-tkcns\") pod \"cert-manager-operator-controller-manager-55c444bf86-wm6n6\" (UID: \"874a9773-1fa4-4640-b50f-574a6e958ce5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wm6n6" Jul 06 11:34:24.765264 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:24.765130 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/874a9773-1fa4-4640-b50f-574a6e958ce5-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wm6n6\" (UID: \"874a9773-1fa4-4640-b50f-574a6e958ce5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wm6n6" Jul 06 11:34:24.765454 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:24.765439 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/874a9773-1fa4-4640-b50f-574a6e958ce5-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wm6n6\" (UID: \"874a9773-1fa4-4640-b50f-574a6e958ce5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wm6n6" Jul 06 11:34:24.775084 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:24.775063 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkcns\" (UniqueName: \"kubernetes.io/projected/874a9773-1fa4-4640-b50f-574a6e958ce5-kube-api-access-tkcns\") pod \"cert-manager-operator-controller-manager-55c444bf86-wm6n6\" (UID: \"874a9773-1fa4-4640-b50f-574a6e958ce5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wm6n6" Jul 06 11:34:24.903442 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:24.903409 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wm6n6" Jul 06 11:34:25.037386 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:25.037360 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wm6n6"] Jul 06 11:34:25.040133 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:34:25.040091 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod874a9773_1fa4_4640_b50f_574a6e958ce5.slice/crio-5cd1e6eb3170395f5f3ae8150e00c859862620dd590bbeb3e985ffc72bb6be34 WatchSource:0}: Error finding container 5cd1e6eb3170395f5f3ae8150e00c859862620dd590bbeb3e985ffc72bb6be34: Status 404 returned error can't find the container with id 5cd1e6eb3170395f5f3ae8150e00c859862620dd590bbeb3e985ffc72bb6be34 Jul 06 11:34:25.367502 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:25.367410 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wm6n6" event={"ID":"874a9773-1fa4-4640-b50f-574a6e958ce5","Type":"ContainerStarted","Data":"5cd1e6eb3170395f5f3ae8150e00c859862620dd590bbeb3e985ffc72bb6be34"} Jul 06 11:34:27.377121 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:27.377088 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wm6n6" event={"ID":"874a9773-1fa4-4640-b50f-574a6e958ce5","Type":"ContainerStarted","Data":"34785d9b10fe70b701c6fa7c80662ecc57f9b600a8c6496c0bbd663121792855"} Jul 06 11:34:27.399443 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:27.399381 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wm6n6" podStartSLOduration=1.9939186549999999 podStartE2EDuration="3.399358508s" totalImagesPullingTime="1.405439853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:34:27.397112878 +0000 UTC m=+365.876911894" watchObservedRunningTime="2026-07-06 11:34:27.399358508 +0000 UTC m=+365.879157529" Jul 06 11:34:28.407242 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:28.407207 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5hwsb"] Jul 06 11:34:28.410164 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:28.410147 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5hwsb" Jul 06 11:34:28.413214 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:28.413190 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-74gpw\"" Jul 06 11:34:28.413214 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:28.413213 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 06 11:34:28.413372 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:28.413190 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 06 11:34:28.422809 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:28.422784 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5hwsb"] Jul 06 11:34:28.490433 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:28.490395 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm6qv\" (UniqueName: \"kubernetes.io/projected/1b0f6518-3f75-4998-a02b-81e8a8fd9b07-kube-api-access-rm6qv\") pod \"cert-manager-webhook-6486ffd997-5hwsb\" (UID: \"1b0f6518-3f75-4998-a02b-81e8a8fd9b07\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5hwsb" Jul 06 11:34:28.490594 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:28.490468 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b0f6518-3f75-4998-a02b-81e8a8fd9b07-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5hwsb\" (UID: \"1b0f6518-3f75-4998-a02b-81e8a8fd9b07\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5hwsb" Jul 06 11:34:28.591789 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:28.591748 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b0f6518-3f75-4998-a02b-81e8a8fd9b07-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5hwsb\" (UID: \"1b0f6518-3f75-4998-a02b-81e8a8fd9b07\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5hwsb" Jul 06 11:34:28.591989 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:28.591806 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm6qv\" (UniqueName: \"kubernetes.io/projected/1b0f6518-3f75-4998-a02b-81e8a8fd9b07-kube-api-access-rm6qv\") pod \"cert-manager-webhook-6486ffd997-5hwsb\" (UID: \"1b0f6518-3f75-4998-a02b-81e8a8fd9b07\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5hwsb" Jul 06 11:34:28.600645 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:28.600610 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b0f6518-3f75-4998-a02b-81e8a8fd9b07-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5hwsb\" (UID: \"1b0f6518-3f75-4998-a02b-81e8a8fd9b07\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5hwsb" Jul 06 11:34:28.600841 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:28.600820 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm6qv\" (UniqueName: \"kubernetes.io/projected/1b0f6518-3f75-4998-a02b-81e8a8fd9b07-kube-api-access-rm6qv\") pod \"cert-manager-webhook-6486ffd997-5hwsb\" (UID: \"1b0f6518-3f75-4998-a02b-81e8a8fd9b07\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5hwsb" Jul 06 11:34:28.718701 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:28.718591 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5hwsb" Jul 06 11:34:28.722389 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:28.722369 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2g9b"] Jul 06 11:34:28.725606 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:28.725589 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2g9b" Jul 06 11:34:28.730990 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:28.730965 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 11:34:28.731117 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:28.730974 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 11:34:28.733591 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:28.733571 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xkzgz\"" Jul 06 11:34:28.745468 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:28.745440 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2g9b"] Jul 06 11:34:28.793865 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:28.793832 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfq7l\" (UniqueName: \"kubernetes.io/projected/3748814e-d081-4592-9b75-443527d99c00-kube-api-access-tfq7l\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2g9b\" (UID: \"3748814e-d081-4592-9b75-443527d99c00\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2g9b" Jul 06 11:34:28.794036 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:28.793887 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3748814e-d081-4592-9b75-443527d99c00-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2g9b\" (UID: \"3748814e-d081-4592-9b75-443527d99c00\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2g9b" Jul 06 11:34:28.794036 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:28.793973 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3748814e-d081-4592-9b75-443527d99c00-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2g9b\" (UID: \"3748814e-d081-4592-9b75-443527d99c00\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2g9b" Jul 06 11:34:28.852249 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:28.852216 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5hwsb"] Jul 06 11:34:28.855028 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:34:28.855001 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b0f6518_3f75_4998_a02b_81e8a8fd9b07.slice/crio-90c8e8f0c27b3a16a6c1684a0b72d3c67324e80aa7e95f2e161d0bda9f396453 WatchSource:0}: Error finding container 90c8e8f0c27b3a16a6c1684a0b72d3c67324e80aa7e95f2e161d0bda9f396453: Status 404 returned error can't find the container with id 90c8e8f0c27b3a16a6c1684a0b72d3c67324e80aa7e95f2e161d0bda9f396453 Jul 06 11:34:28.894570 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:28.894528 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3748814e-d081-4592-9b75-443527d99c00-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2g9b\" (UID: \"3748814e-d081-4592-9b75-443527d99c00\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2g9b" Jul 06 11:34:28.894753 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:28.894588 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfq7l\" (UniqueName: \"kubernetes.io/projected/3748814e-d081-4592-9b75-443527d99c00-kube-api-access-tfq7l\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2g9b\" (UID: \"3748814e-d081-4592-9b75-443527d99c00\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2g9b" Jul 06 11:34:28.894753 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:28.894636 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3748814e-d081-4592-9b75-443527d99c00-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2g9b\" (UID: \"3748814e-d081-4592-9b75-443527d99c00\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2g9b" Jul 06 11:34:28.894956 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:28.894936 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3748814e-d081-4592-9b75-443527d99c00-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2g9b\" (UID: \"3748814e-d081-4592-9b75-443527d99c00\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2g9b" Jul 06 11:34:28.895010 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:28.894993 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3748814e-d081-4592-9b75-443527d99c00-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2g9b\" (UID: \"3748814e-d081-4592-9b75-443527d99c00\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2g9b" Jul 06 11:34:28.906202 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:28.906178 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfq7l\" (UniqueName: \"kubernetes.io/projected/3748814e-d081-4592-9b75-443527d99c00-kube-api-access-tfq7l\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2g9b\" (UID: \"3748814e-d081-4592-9b75-443527d99c00\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2g9b" Jul 06 11:34:29.046433 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:29.046352 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2g9b" Jul 06 11:34:29.174484 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:29.174407 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2g9b"] Jul 06 11:34:29.176917 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:34:29.176880 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3748814e_d081_4592_9b75_443527d99c00.slice/crio-f12ea971a674b76b118b486b5e6d1810a95e1d0a4732700556d0d1748573b15b WatchSource:0}: Error finding container f12ea971a674b76b118b486b5e6d1810a95e1d0a4732700556d0d1748573b15b: Status 404 returned error can't find the container with id f12ea971a674b76b118b486b5e6d1810a95e1d0a4732700556d0d1748573b15b Jul 06 11:34:29.386732 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:29.386695 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5hwsb" event={"ID":"1b0f6518-3f75-4998-a02b-81e8a8fd9b07","Type":"ContainerStarted","Data":"90c8e8f0c27b3a16a6c1684a0b72d3c67324e80aa7e95f2e161d0bda9f396453"} Jul 06 11:34:29.388090 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:29.388058 2568 generic.go:356] "Generic (PLEG): container finished" podID="3748814e-d081-4592-9b75-443527d99c00" containerID="8f97f9150ec005fa330aeb82976b3ed73ebf7785bbb89088b685afbc5bcdefe5" exitCode=0 Jul 06 11:34:29.388219 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:29.388131 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2g9b" event={"ID":"3748814e-d081-4592-9b75-443527d99c00","Type":"ContainerDied","Data":"8f97f9150ec005fa330aeb82976b3ed73ebf7785bbb89088b685afbc5bcdefe5"} Jul 06 11:34:29.388219 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:29.388165 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2g9b" event={"ID":"3748814e-d081-4592-9b75-443527d99c00","Type":"ContainerStarted","Data":"f12ea971a674b76b118b486b5e6d1810a95e1d0a4732700556d0d1748573b15b"} Jul 06 11:34:32.402495 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:32.402384 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5hwsb" event={"ID":"1b0f6518-3f75-4998-a02b-81e8a8fd9b07","Type":"ContainerStarted","Data":"baa8c712a2570d8e52b7965390ef5ce6a995821fd65644258a30286410757989"} Jul 06 11:34:32.402495 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:32.402442 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5hwsb" Jul 06 11:34:32.404103 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:32.404075 2568 generic.go:356] "Generic (PLEG): container finished" podID="3748814e-d081-4592-9b75-443527d99c00" containerID="78e7c85ecac2ce15440569c4568c62a6abb6588db508919f1329ff172ca07447" exitCode=0 Jul 06 11:34:32.404230 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:32.404122 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2g9b" event={"ID":"3748814e-d081-4592-9b75-443527d99c00","Type":"ContainerDied","Data":"78e7c85ecac2ce15440569c4568c62a6abb6588db508919f1329ff172ca07447"} Jul 06 11:34:32.421770 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:32.421722 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-5hwsb" podStartSLOduration=1.126225399 podStartE2EDuration="4.421704264s" totalImagesPullingTime="3.295478865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:34:32.419301249 +0000 UTC m=+370.899100269" watchObservedRunningTime="2026-07-06 11:34:32.421704264 +0000 UTC m=+370.901503284" Jul 06 11:34:33.410717 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:33.410659 2568 generic.go:356] "Generic (PLEG): container finished" podID="3748814e-d081-4592-9b75-443527d99c00" containerID="fd8157a3c05d27bdf0d300162abf23c40812adef355cccfc4bcbf242a5567407" exitCode=0 Jul 06 11:34:33.411206 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:33.410745 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2g9b" event={"ID":"3748814e-d081-4592-9b75-443527d99c00","Type":"ContainerDied","Data":"fd8157a3c05d27bdf0d300162abf23c40812adef355cccfc4bcbf242a5567407"} Jul 06 11:34:34.540823 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:34.540799 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2g9b" Jul 06 11:34:34.645428 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:34.645392 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3748814e-d081-4592-9b75-443527d99c00-bundle\") pod \"3748814e-d081-4592-9b75-443527d99c00\" (UID: \"3748814e-d081-4592-9b75-443527d99c00\") " Jul 06 11:34:34.645619 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:34.645458 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfq7l\" (UniqueName: \"kubernetes.io/projected/3748814e-d081-4592-9b75-443527d99c00-kube-api-access-tfq7l\") pod \"3748814e-d081-4592-9b75-443527d99c00\" (UID: \"3748814e-d081-4592-9b75-443527d99c00\") " Jul 06 11:34:34.645619 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:34.645490 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3748814e-d081-4592-9b75-443527d99c00-util\") pod \"3748814e-d081-4592-9b75-443527d99c00\" (UID: \"3748814e-d081-4592-9b75-443527d99c00\") " Jul 06 11:34:34.645851 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:34.645818 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3748814e-d081-4592-9b75-443527d99c00-bundle" (OuterVolumeSpecName: "bundle") pod "3748814e-d081-4592-9b75-443527d99c00" (UID: "3748814e-d081-4592-9b75-443527d99c00"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:34:34.647506 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:34.647482 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3748814e-d081-4592-9b75-443527d99c00-kube-api-access-tfq7l" (OuterVolumeSpecName: "kube-api-access-tfq7l") pod "3748814e-d081-4592-9b75-443527d99c00" (UID: "3748814e-d081-4592-9b75-443527d99c00"). InnerVolumeSpecName "kube-api-access-tfq7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 11:34:34.650247 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:34.650203 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3748814e-d081-4592-9b75-443527d99c00-util" (OuterVolumeSpecName: "util") pod "3748814e-d081-4592-9b75-443527d99c00" (UID: "3748814e-d081-4592-9b75-443527d99c00"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:34:34.746044 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:34.745944 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfq7l\" (UniqueName: \"kubernetes.io/projected/3748814e-d081-4592-9b75-443527d99c00-kube-api-access-tfq7l\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:34:34.746044 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:34.745979 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3748814e-d081-4592-9b75-443527d99c00-util\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:34:34.746044 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:34.745994 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3748814e-d081-4592-9b75-443527d99c00-bundle\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:34:35.421738 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:35.421704 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2g9b" event={"ID":"3748814e-d081-4592-9b75-443527d99c00","Type":"ContainerDied","Data":"f12ea971a674b76b118b486b5e6d1810a95e1d0a4732700556d0d1748573b15b"} Jul 06 11:34:35.421738 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:35.421725 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2g9b" Jul 06 11:34:35.421950 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:35.421756 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f12ea971a674b76b118b486b5e6d1810a95e1d0a4732700556d0d1748573b15b" Jul 06 11:34:38.413219 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:38.413186 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5hwsb" Jul 06 11:34:42.126714 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:42.126662 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lwb7h"] Jul 06 11:34:42.127117 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:42.126936 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3748814e-d081-4592-9b75-443527d99c00" containerName="util" Jul 06 11:34:42.127117 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:42.126948 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3748814e-d081-4592-9b75-443527d99c00" containerName="util" Jul 06 11:34:42.127117 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:42.126957 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3748814e-d081-4592-9b75-443527d99c00" containerName="extract" Jul 06 11:34:42.127117 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:42.126963 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3748814e-d081-4592-9b75-443527d99c00" containerName="extract" Jul 06 11:34:42.127117 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:42.126983 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3748814e-d081-4592-9b75-443527d99c00" containerName="pull" Jul 06 11:34:42.127117 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:42.126989 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3748814e-d081-4592-9b75-443527d99c00" containerName="pull" Jul 06 11:34:42.127117 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:42.127028 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3748814e-d081-4592-9b75-443527d99c00" containerName="extract" Jul 06 11:34:42.168241 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:42.168198 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lwb7h"] Jul 06 11:34:42.168401 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:42.168311 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lwb7h" Jul 06 11:34:42.174634 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:42.174556 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 11:34:42.174743 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:42.174660 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-pmw7m\"" Jul 06 11:34:42.175661 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:42.175647 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 11:34:42.200807 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:42.200775 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cds26\" (UniqueName: \"kubernetes.io/projected/1be8510e-b7b9-4c5b-93ba-5c14cb7392e0-kube-api-access-cds26\") pod \"openshift-lws-operator-fd8ccff4c-lwb7h\" (UID: \"1be8510e-b7b9-4c5b-93ba-5c14cb7392e0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lwb7h" Jul 06 11:34:42.200975 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:42.200841 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1be8510e-b7b9-4c5b-93ba-5c14cb7392e0-tmp\") pod \"openshift-lws-operator-fd8ccff4c-lwb7h\" (UID: \"1be8510e-b7b9-4c5b-93ba-5c14cb7392e0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lwb7h" Jul 06 11:34:42.301451 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:42.301414 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1be8510e-b7b9-4c5b-93ba-5c14cb7392e0-tmp\") pod \"openshift-lws-operator-fd8ccff4c-lwb7h\" (UID: \"1be8510e-b7b9-4c5b-93ba-5c14cb7392e0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lwb7h" Jul 06 11:34:42.301621 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:42.301463 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cds26\" (UniqueName: \"kubernetes.io/projected/1be8510e-b7b9-4c5b-93ba-5c14cb7392e0-kube-api-access-cds26\") pod \"openshift-lws-operator-fd8ccff4c-lwb7h\" (UID: \"1be8510e-b7b9-4c5b-93ba-5c14cb7392e0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lwb7h" Jul 06 11:34:42.301845 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:42.301821 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1be8510e-b7b9-4c5b-93ba-5c14cb7392e0-tmp\") pod \"openshift-lws-operator-fd8ccff4c-lwb7h\" (UID: \"1be8510e-b7b9-4c5b-93ba-5c14cb7392e0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lwb7h" Jul 06 11:34:42.311849 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:42.311823 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cds26\" (UniqueName: \"kubernetes.io/projected/1be8510e-b7b9-4c5b-93ba-5c14cb7392e0-kube-api-access-cds26\") pod \"openshift-lws-operator-fd8ccff4c-lwb7h\" (UID: \"1be8510e-b7b9-4c5b-93ba-5c14cb7392e0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lwb7h" Jul 06 11:34:42.477600 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:42.477569 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lwb7h" Jul 06 11:34:42.633457 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:42.633430 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lwb7h"] Jul 06 11:34:42.636084 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:34:42.636038 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1be8510e_b7b9_4c5b_93ba_5c14cb7392e0.slice/crio-cf16d0efa373f201cd0a59ca70ef25dad5ed2182d39f19f19f31ff1e1762edbf WatchSource:0}: Error finding container cf16d0efa373f201cd0a59ca70ef25dad5ed2182d39f19f19f31ff1e1762edbf: Status 404 returned error can't find the container with id cf16d0efa373f201cd0a59ca70ef25dad5ed2182d39f19f19f31ff1e1762edbf Jul 06 11:34:43.453344 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:43.453308 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lwb7h" event={"ID":"1be8510e-b7b9-4c5b-93ba-5c14cb7392e0","Type":"ContainerStarted","Data":"cf16d0efa373f201cd0a59ca70ef25dad5ed2182d39f19f19f31ff1e1762edbf"} Jul 06 11:34:45.463237 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:45.463198 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lwb7h" event={"ID":"1be8510e-b7b9-4c5b-93ba-5c14cb7392e0","Type":"ContainerStarted","Data":"a7b375c48fb3a68fe2ad18eecab3d4cb618f4e7f323cc82f44101ac203410b3c"} Jul 06 11:34:45.483090 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:45.483031 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lwb7h" podStartSLOduration=1.106878989 podStartE2EDuration="3.483014886s" totalImagesPullingTime="2.376135897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:34:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:34:45.481829296 +0000 UTC m=+383.961628312" watchObservedRunningTime="2026-07-06 11:34:45.483014886 +0000 UTC m=+383.962813898" Jul 06 11:34:47.993258 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:47.993219 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpcqpz"] Jul 06 11:34:48.014447 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:48.014417 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpcqpz"] Jul 06 11:34:48.014586 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:48.014523 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpcqpz" Jul 06 11:34:48.017628 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:48.017603 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 11:34:48.018102 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:48.018082 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 11:34:48.018968 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:48.018950 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xkzgz\"" Jul 06 11:34:48.051419 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:48.051392 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7496f3bd-10e7-4c9b-83f0-64e376418df4-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpcqpz\" (UID: \"7496f3bd-10e7-4c9b-83f0-64e376418df4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpcqpz" Jul 06 11:34:48.051590 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:48.051446 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7496f3bd-10e7-4c9b-83f0-64e376418df4-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpcqpz\" (UID: \"7496f3bd-10e7-4c9b-83f0-64e376418df4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpcqpz" Jul 06 11:34:48.051590 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:48.051489 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9trnx\" (UniqueName: \"kubernetes.io/projected/7496f3bd-10e7-4c9b-83f0-64e376418df4-kube-api-access-9trnx\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpcqpz\" (UID: \"7496f3bd-10e7-4c9b-83f0-64e376418df4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpcqpz" Jul 06 11:34:48.152482 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:48.152448 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7496f3bd-10e7-4c9b-83f0-64e376418df4-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpcqpz\" (UID: \"7496f3bd-10e7-4c9b-83f0-64e376418df4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpcqpz" Jul 06 11:34:48.152695 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:48.152494 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7496f3bd-10e7-4c9b-83f0-64e376418df4-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpcqpz\" (UID: \"7496f3bd-10e7-4c9b-83f0-64e376418df4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpcqpz" Jul 06 11:34:48.152695 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:48.152517 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9trnx\" (UniqueName: \"kubernetes.io/projected/7496f3bd-10e7-4c9b-83f0-64e376418df4-kube-api-access-9trnx\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpcqpz\" (UID: \"7496f3bd-10e7-4c9b-83f0-64e376418df4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpcqpz" Jul 06 11:34:48.152855 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:48.152833 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7496f3bd-10e7-4c9b-83f0-64e376418df4-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpcqpz\" (UID: \"7496f3bd-10e7-4c9b-83f0-64e376418df4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpcqpz" Jul 06 11:34:48.152931 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:48.152891 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7496f3bd-10e7-4c9b-83f0-64e376418df4-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpcqpz\" (UID: \"7496f3bd-10e7-4c9b-83f0-64e376418df4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpcqpz" Jul 06 11:34:48.167107 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:48.167074 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9trnx\" (UniqueName: \"kubernetes.io/projected/7496f3bd-10e7-4c9b-83f0-64e376418df4-kube-api-access-9trnx\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpcqpz\" (UID: \"7496f3bd-10e7-4c9b-83f0-64e376418df4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpcqpz" Jul 06 11:34:48.323950 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:48.323846 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpcqpz" Jul 06 11:34:48.452291 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:48.452266 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpcqpz"] Jul 06 11:34:48.454892 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:34:48.454850 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7496f3bd_10e7_4c9b_83f0_64e376418df4.slice/crio-0dcf52d6dd01d4e55f073e208cc118a01945fb15bb74e76d2f714019d552a60f WatchSource:0}: Error finding container 0dcf52d6dd01d4e55f073e208cc118a01945fb15bb74e76d2f714019d552a60f: Status 404 returned error can't find the container with id 0dcf52d6dd01d4e55f073e208cc118a01945fb15bb74e76d2f714019d552a60f Jul 06 11:34:48.477579 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:48.477549 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpcqpz" event={"ID":"7496f3bd-10e7-4c9b-83f0-64e376418df4","Type":"ContainerStarted","Data":"0dcf52d6dd01d4e55f073e208cc118a01945fb15bb74e76d2f714019d552a60f"} Jul 06 11:34:49.482770 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:49.482658 2568 generic.go:356] "Generic (PLEG): container finished" podID="7496f3bd-10e7-4c9b-83f0-64e376418df4" containerID="8aabd42efb8c4d85919ef154bf6c9f2bcba6903de2c7835550dbef2a694f8991" exitCode=0 Jul 06 11:34:49.482770 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:49.482744 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpcqpz" event={"ID":"7496f3bd-10e7-4c9b-83f0-64e376418df4","Type":"ContainerDied","Data":"8aabd42efb8c4d85919ef154bf6c9f2bcba6903de2c7835550dbef2a694f8991"} Jul 06 11:34:51.494234 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:51.494199 2568 generic.go:356] "Generic (PLEG): container finished" podID="7496f3bd-10e7-4c9b-83f0-64e376418df4" containerID="fb7f1eefadf610bd609478e3f69dcda7e5d371c75a3d472f68067aaa1f45ae56" exitCode=0 Jul 06 11:34:51.494631 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:51.494265 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpcqpz" event={"ID":"7496f3bd-10e7-4c9b-83f0-64e376418df4","Type":"ContainerDied","Data":"fb7f1eefadf610bd609478e3f69dcda7e5d371c75a3d472f68067aaa1f45ae56"} Jul 06 11:34:52.501072 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:52.501037 2568 generic.go:356] "Generic (PLEG): container finished" podID="7496f3bd-10e7-4c9b-83f0-64e376418df4" containerID="9fb23a380241f45e7e5e02f34d9d065fa647c5a0c4e44145ced3dfafec363e9f" exitCode=0 Jul 06 11:34:52.501450 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:52.501115 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpcqpz" event={"ID":"7496f3bd-10e7-4c9b-83f0-64e376418df4","Type":"ContainerDied","Data":"9fb23a380241f45e7e5e02f34d9d065fa647c5a0c4e44145ced3dfafec363e9f"} Jul 06 11:34:53.634478 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:53.634453 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpcqpz" Jul 06 11:34:53.700384 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:53.700350 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9trnx\" (UniqueName: \"kubernetes.io/projected/7496f3bd-10e7-4c9b-83f0-64e376418df4-kube-api-access-9trnx\") pod \"7496f3bd-10e7-4c9b-83f0-64e376418df4\" (UID: \"7496f3bd-10e7-4c9b-83f0-64e376418df4\") " Jul 06 11:34:53.700552 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:53.700388 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7496f3bd-10e7-4c9b-83f0-64e376418df4-bundle\") pod \"7496f3bd-10e7-4c9b-83f0-64e376418df4\" (UID: \"7496f3bd-10e7-4c9b-83f0-64e376418df4\") " Jul 06 11:34:53.700552 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:53.700454 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7496f3bd-10e7-4c9b-83f0-64e376418df4-util\") pod \"7496f3bd-10e7-4c9b-83f0-64e376418df4\" (UID: \"7496f3bd-10e7-4c9b-83f0-64e376418df4\") " Jul 06 11:34:53.701144 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:53.701116 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7496f3bd-10e7-4c9b-83f0-64e376418df4-bundle" (OuterVolumeSpecName: "bundle") pod "7496f3bd-10e7-4c9b-83f0-64e376418df4" (UID: "7496f3bd-10e7-4c9b-83f0-64e376418df4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:34:53.702511 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:53.702483 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7496f3bd-10e7-4c9b-83f0-64e376418df4-kube-api-access-9trnx" (OuterVolumeSpecName: "kube-api-access-9trnx") pod "7496f3bd-10e7-4c9b-83f0-64e376418df4" (UID: "7496f3bd-10e7-4c9b-83f0-64e376418df4"). InnerVolumeSpecName "kube-api-access-9trnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 11:34:53.705811 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:53.705789 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7496f3bd-10e7-4c9b-83f0-64e376418df4-util" (OuterVolumeSpecName: "util") pod "7496f3bd-10e7-4c9b-83f0-64e376418df4" (UID: "7496f3bd-10e7-4c9b-83f0-64e376418df4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:34:53.801707 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:53.801593 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7496f3bd-10e7-4c9b-83f0-64e376418df4-util\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:34:53.801707 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:53.801626 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9trnx\" (UniqueName: \"kubernetes.io/projected/7496f3bd-10e7-4c9b-83f0-64e376418df4-kube-api-access-9trnx\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:34:53.801707 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:53.801637 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7496f3bd-10e7-4c9b-83f0-64e376418df4-bundle\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:34:54.510696 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:54.510650 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpcqpz" event={"ID":"7496f3bd-10e7-4c9b-83f0-64e376418df4","Type":"ContainerDied","Data":"0dcf52d6dd01d4e55f073e208cc118a01945fb15bb74e76d2f714019d552a60f"} Jul 06 11:34:54.510866 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:54.510712 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0dcf52d6dd01d4e55f073e208cc118a01945fb15bb74e76d2f714019d552a60f" Jul 06 11:34:54.510866 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:34:54.510664 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpcqpz" Jul 06 11:35:04.663074 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:04.663037 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cb7946974-kcmtw"] Jul 06 11:35:04.663502 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:04.663284 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7496f3bd-10e7-4c9b-83f0-64e376418df4" containerName="util" Jul 06 11:35:04.663502 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:04.663295 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7496f3bd-10e7-4c9b-83f0-64e376418df4" containerName="util" Jul 06 11:35:04.663502 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:04.663305 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7496f3bd-10e7-4c9b-83f0-64e376418df4" containerName="pull" Jul 06 11:35:04.663502 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:04.663311 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7496f3bd-10e7-4c9b-83f0-64e376418df4" containerName="pull" Jul 06 11:35:04.663502 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:04.663325 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7496f3bd-10e7-4c9b-83f0-64e376418df4" containerName="extract" Jul 06 11:35:04.663502 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:04.663330 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7496f3bd-10e7-4c9b-83f0-64e376418df4" containerName="extract" Jul 06 11:35:04.663502 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:04.663376 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7496f3bd-10e7-4c9b-83f0-64e376418df4" containerName="extract" Jul 06 11:35:04.667441 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:04.667419 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cb7946974-kcmtw" Jul 06 11:35:04.670249 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:04.670226 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 11:35:04.670403 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:04.670384 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 11:35:04.670452 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:04.670420 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 11:35:04.670638 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:04.670621 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 11:35:04.670750 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:04.670705 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dmxtp\"" Jul 06 11:35:04.681015 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:04.680989 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cb7946974-kcmtw"] Jul 06 11:35:04.786735 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:04.786696 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ce314f78-7470-45c3-9a73-cede06d44b62-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cb7946974-kcmtw\" (UID: \"ce314f78-7470-45c3-9a73-cede06d44b62\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb7946974-kcmtw" Jul 06 11:35:04.786917 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:04.786753 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ce314f78-7470-45c3-9a73-cede06d44b62-manifests\") pod \"opendatahub-operator-controller-manager-6cb7946974-kcmtw\" (UID: \"ce314f78-7470-45c3-9a73-cede06d44b62\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb7946974-kcmtw" Jul 06 11:35:04.786917 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:04.786779 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce314f78-7470-45c3-9a73-cede06d44b62-tmp\") pod \"opendatahub-operator-controller-manager-6cb7946974-kcmtw\" (UID: \"ce314f78-7470-45c3-9a73-cede06d44b62\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb7946974-kcmtw" Jul 06 11:35:04.786917 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:04.786826 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzvcj\" (UniqueName: \"kubernetes.io/projected/ce314f78-7470-45c3-9a73-cede06d44b62-kube-api-access-fzvcj\") pod \"opendatahub-operator-controller-manager-6cb7946974-kcmtw\" (UID: \"ce314f78-7470-45c3-9a73-cede06d44b62\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb7946974-kcmtw" Jul 06 11:35:04.786917 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:04.786862 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce314f78-7470-45c3-9a73-cede06d44b62-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cb7946974-kcmtw\" (UID: \"ce314f78-7470-45c3-9a73-cede06d44b62\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb7946974-kcmtw" Jul 06 11:35:04.887293 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:04.887234 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ce314f78-7470-45c3-9a73-cede06d44b62-manifests\") pod \"opendatahub-operator-controller-manager-6cb7946974-kcmtw\" (UID: \"ce314f78-7470-45c3-9a73-cede06d44b62\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb7946974-kcmtw" Jul 06 11:35:04.887293 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:04.887296 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce314f78-7470-45c3-9a73-cede06d44b62-tmp\") pod \"opendatahub-operator-controller-manager-6cb7946974-kcmtw\" (UID: \"ce314f78-7470-45c3-9a73-cede06d44b62\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb7946974-kcmtw" Jul 06 11:35:04.887571 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:04.887325 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzvcj\" (UniqueName: \"kubernetes.io/projected/ce314f78-7470-45c3-9a73-cede06d44b62-kube-api-access-fzvcj\") pod \"opendatahub-operator-controller-manager-6cb7946974-kcmtw\" (UID: \"ce314f78-7470-45c3-9a73-cede06d44b62\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb7946974-kcmtw" Jul 06 11:35:04.887571 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:04.887364 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce314f78-7470-45c3-9a73-cede06d44b62-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cb7946974-kcmtw\" (UID: \"ce314f78-7470-45c3-9a73-cede06d44b62\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb7946974-kcmtw" Jul 06 11:35:04.887571 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:04.887385 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ce314f78-7470-45c3-9a73-cede06d44b62-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cb7946974-kcmtw\" (UID: \"ce314f78-7470-45c3-9a73-cede06d44b62\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb7946974-kcmtw" Jul 06 11:35:04.887759 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:04.887642 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ce314f78-7470-45c3-9a73-cede06d44b62-manifests\") pod \"opendatahub-operator-controller-manager-6cb7946974-kcmtw\" (UID: \"ce314f78-7470-45c3-9a73-cede06d44b62\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb7946974-kcmtw" Jul 06 11:35:04.889625 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:04.889592 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce314f78-7470-45c3-9a73-cede06d44b62-tmp\") pod \"opendatahub-operator-controller-manager-6cb7946974-kcmtw\" (UID: \"ce314f78-7470-45c3-9a73-cede06d44b62\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb7946974-kcmtw" Jul 06 11:35:04.889858 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:04.889840 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ce314f78-7470-45c3-9a73-cede06d44b62-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cb7946974-kcmtw\" (UID: \"ce314f78-7470-45c3-9a73-cede06d44b62\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb7946974-kcmtw" Jul 06 11:35:04.889940 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:04.889921 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce314f78-7470-45c3-9a73-cede06d44b62-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cb7946974-kcmtw\" (UID: \"ce314f78-7470-45c3-9a73-cede06d44b62\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb7946974-kcmtw" Jul 06 11:35:04.897586 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:04.897557 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzvcj\" (UniqueName: \"kubernetes.io/projected/ce314f78-7470-45c3-9a73-cede06d44b62-kube-api-access-fzvcj\") pod \"opendatahub-operator-controller-manager-6cb7946974-kcmtw\" (UID: \"ce314f78-7470-45c3-9a73-cede06d44b62\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb7946974-kcmtw" Jul 06 11:35:04.978579 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:04.978469 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cb7946974-kcmtw" Jul 06 11:35:05.121554 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:05.121515 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cb7946974-kcmtw"] Jul 06 11:35:05.124728 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:35:05.124700 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce314f78_7470_45c3_9a73_cede06d44b62.slice/crio-01941f81ed03ef491d481f45405eaa49c2ce19aaec8a432d6b21a0febc8e9650 WatchSource:0}: Error finding container 01941f81ed03ef491d481f45405eaa49c2ce19aaec8a432d6b21a0febc8e9650: Status 404 returned error can't find the container with id 01941f81ed03ef491d481f45405eaa49c2ce19aaec8a432d6b21a0febc8e9650 Jul 06 11:35:05.558451 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:05.558416 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cb7946974-kcmtw" event={"ID":"ce314f78-7470-45c3-9a73-cede06d44b62","Type":"ContainerStarted","Data":"01941f81ed03ef491d481f45405eaa49c2ce19aaec8a432d6b21a0febc8e9650"} Jul 06 11:35:08.575653 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:08.575614 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cb7946974-kcmtw" event={"ID":"ce314f78-7470-45c3-9a73-cede06d44b62","Type":"ContainerStarted","Data":"079136f3ed0dd2b82504ce500f1768433f1353b508644acce84f856b0bfb5ca0"} Jul 06 11:35:09.581807 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:09.581772 2568 generic.go:356] "Generic (PLEG): container finished" podID="ce314f78-7470-45c3-9a73-cede06d44b62" containerID="079136f3ed0dd2b82504ce500f1768433f1353b508644acce84f856b0bfb5ca0" exitCode=0 Jul 06 11:35:09.582203 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:09.581839 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cb7946974-kcmtw" event={"ID":"ce314f78-7470-45c3-9a73-cede06d44b62","Type":"ContainerDied","Data":"079136f3ed0dd2b82504ce500f1768433f1353b508644acce84f856b0bfb5ca0"} Jul 06 11:35:10.588206 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:10.588160 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cb7946974-kcmtw" event={"ID":"ce314f78-7470-45c3-9a73-cede06d44b62","Type":"ContainerStarted","Data":"c07f7891df22c6572490e1417da22b535580759e471ac214801b0c3f7b081de3"} Jul 06 11:35:10.588671 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:10.588308 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cb7946974-kcmtw" Jul 06 11:35:10.618094 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:10.617993 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6cb7946974-kcmtw" podStartSLOduration=1.449882425 podStartE2EDuration="6.617974881s" totalImagesPullingTime="4.111309562s" totalInitContainerRuntime="1.056782894s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:35:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:35:10.615611006 +0000 UTC m=+409.095410048" watchObservedRunningTime="2026-07-06 11:35:10.617974881 +0000 UTC m=+409.097773897" Jul 06 11:35:21.595045 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:21.595014 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6cb7946974-kcmtw" Jul 06 11:35:25.825723 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:25.825670 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-kdsrp"] Jul 06 11:35:25.830494 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:25.830467 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-kdsrp" Jul 06 11:35:25.833277 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:25.833255 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 06 11:35:25.833386 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:25.833321 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-n8wf5\"" Jul 06 11:35:25.842133 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:25.842113 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-kdsrp"] Jul 06 11:35:25.857885 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:25.857846 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdqgw\" (UniqueName: \"kubernetes.io/projected/6bfe9a65-bdfc-431b-b376-40f507daf37a-kube-api-access-tdqgw\") pod \"odh-model-controller-7998c67dc4-kdsrp\" (UID: \"6bfe9a65-bdfc-431b-b376-40f507daf37a\") " pod="opendatahub/odh-model-controller-7998c67dc4-kdsrp" Jul 06 11:35:25.858031 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:25.857968 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6bfe9a65-bdfc-431b-b376-40f507daf37a-cert\") pod \"odh-model-controller-7998c67dc4-kdsrp\" (UID: \"6bfe9a65-bdfc-431b-b376-40f507daf37a\") " pod="opendatahub/odh-model-controller-7998c67dc4-kdsrp" Jul 06 11:35:25.958667 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:25.958630 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdqgw\" (UniqueName: \"kubernetes.io/projected/6bfe9a65-bdfc-431b-b376-40f507daf37a-kube-api-access-tdqgw\") pod \"odh-model-controller-7998c67dc4-kdsrp\" (UID: \"6bfe9a65-bdfc-431b-b376-40f507daf37a\") " pod="opendatahub/odh-model-controller-7998c67dc4-kdsrp" Jul 06 11:35:25.958971 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:25.958742 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6bfe9a65-bdfc-431b-b376-40f507daf37a-cert\") pod \"odh-model-controller-7998c67dc4-kdsrp\" (UID: \"6bfe9a65-bdfc-431b-b376-40f507daf37a\") " pod="opendatahub/odh-model-controller-7998c67dc4-kdsrp" Jul 06 11:35:25.961278 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:25.961241 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6bfe9a65-bdfc-431b-b376-40f507daf37a-cert\") pod \"odh-model-controller-7998c67dc4-kdsrp\" (UID: \"6bfe9a65-bdfc-431b-b376-40f507daf37a\") " pod="opendatahub/odh-model-controller-7998c67dc4-kdsrp" Jul 06 11:35:25.968169 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:25.968133 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdqgw\" (UniqueName: \"kubernetes.io/projected/6bfe9a65-bdfc-431b-b376-40f507daf37a-kube-api-access-tdqgw\") pod \"odh-model-controller-7998c67dc4-kdsrp\" (UID: \"6bfe9a65-bdfc-431b-b376-40f507daf37a\") " pod="opendatahub/odh-model-controller-7998c67dc4-kdsrp" Jul 06 11:35:26.140088 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:26.140045 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-kdsrp" Jul 06 11:35:26.276370 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:26.276341 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-kdsrp"] Jul 06 11:35:26.278762 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:35:26.278736 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bfe9a65_bdfc_431b_b376_40f507daf37a.slice/crio-55070c4eb7c69c4b609599a8afd4ac6e0a2d2a66d639e01e9bad5e8042e8824f WatchSource:0}: Error finding container 55070c4eb7c69c4b609599a8afd4ac6e0a2d2a66d639e01e9bad5e8042e8824f: Status 404 returned error can't find the container with id 55070c4eb7c69c4b609599a8afd4ac6e0a2d2a66d639e01e9bad5e8042e8824f Jul 06 11:35:26.461747 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:26.461641 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-95wlx"] Jul 06 11:35:26.465943 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:26.465917 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-95wlx" Jul 06 11:35:26.469275 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:26.469250 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 06 11:35:26.477455 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:26.477433 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-95wlx"] Jul 06 11:35:26.563876 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:26.563835 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzzdl\" (UniqueName: \"kubernetes.io/projected/dfcbe27d-5c42-45f9-a719-5b43b710c11d-kube-api-access-nzzdl\") pod \"model-serving-api-54d89bd5fc-95wlx\" (UID: \"dfcbe27d-5c42-45f9-a719-5b43b710c11d\") " pod="opendatahub/model-serving-api-54d89bd5fc-95wlx" Jul 06 11:35:26.564062 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:26.563937 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfcbe27d-5c42-45f9-a719-5b43b710c11d-tls-certs\") pod \"model-serving-api-54d89bd5fc-95wlx\" (UID: \"dfcbe27d-5c42-45f9-a719-5b43b710c11d\") " pod="opendatahub/model-serving-api-54d89bd5fc-95wlx" Jul 06 11:35:26.656470 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:26.656422 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-kdsrp" event={"ID":"6bfe9a65-bdfc-431b-b376-40f507daf37a","Type":"ContainerStarted","Data":"55070c4eb7c69c4b609599a8afd4ac6e0a2d2a66d639e01e9bad5e8042e8824f"} Jul 06 11:35:26.664812 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:26.664779 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfcbe27d-5c42-45f9-a719-5b43b710c11d-tls-certs\") pod \"model-serving-api-54d89bd5fc-95wlx\" (UID: \"dfcbe27d-5c42-45f9-a719-5b43b710c11d\") " pod="opendatahub/model-serving-api-54d89bd5fc-95wlx" Jul 06 11:35:26.664978 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:26.664862 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzzdl\" (UniqueName: \"kubernetes.io/projected/dfcbe27d-5c42-45f9-a719-5b43b710c11d-kube-api-access-nzzdl\") pod \"model-serving-api-54d89bd5fc-95wlx\" (UID: \"dfcbe27d-5c42-45f9-a719-5b43b710c11d\") " pod="opendatahub/model-serving-api-54d89bd5fc-95wlx" Jul 06 11:35:26.664978 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:35:26.664900 2568 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 06 11:35:26.664978 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:35:26.664975 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfcbe27d-5c42-45f9-a719-5b43b710c11d-tls-certs podName:dfcbe27d-5c42-45f9-a719-5b43b710c11d nodeName:}" failed. No retries permitted until 2026-07-06 11:35:27.16495556 +0000 UTC m=+425.644754555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/dfcbe27d-5c42-45f9-a719-5b43b710c11d-tls-certs") pod "model-serving-api-54d89bd5fc-95wlx" (UID: "dfcbe27d-5c42-45f9-a719-5b43b710c11d") : secret "model-serving-api-tls" not found Jul 06 11:35:26.677227 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:26.677198 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzzdl\" (UniqueName: \"kubernetes.io/projected/dfcbe27d-5c42-45f9-a719-5b43b710c11d-kube-api-access-nzzdl\") pod \"model-serving-api-54d89bd5fc-95wlx\" (UID: \"dfcbe27d-5c42-45f9-a719-5b43b710c11d\") " pod="opendatahub/model-serving-api-54d89bd5fc-95wlx" Jul 06 11:35:27.168773 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:27.168666 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfcbe27d-5c42-45f9-a719-5b43b710c11d-tls-certs\") pod \"model-serving-api-54d89bd5fc-95wlx\" (UID: \"dfcbe27d-5c42-45f9-a719-5b43b710c11d\") " pod="opendatahub/model-serving-api-54d89bd5fc-95wlx" Jul 06 11:35:27.171559 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:27.171532 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfcbe27d-5c42-45f9-a719-5b43b710c11d-tls-certs\") pod \"model-serving-api-54d89bd5fc-95wlx\" (UID: \"dfcbe27d-5c42-45f9-a719-5b43b710c11d\") " pod="opendatahub/model-serving-api-54d89bd5fc-95wlx" Jul 06 11:35:27.377305 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:27.377270 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-95wlx" Jul 06 11:35:27.539960 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:27.539846 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-95wlx"] Jul 06 11:35:27.545879 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:35:27.545825 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfcbe27d_5c42_45f9_a719_5b43b710c11d.slice/crio-c2a5a24b43b0b6ec62e447fe0d0a1277bd772b800a1b3e5b1b2cf62aa8f89c49 WatchSource:0}: Error finding container c2a5a24b43b0b6ec62e447fe0d0a1277bd772b800a1b3e5b1b2cf62aa8f89c49: Status 404 returned error can't find the container with id c2a5a24b43b0b6ec62e447fe0d0a1277bd772b800a1b3e5b1b2cf62aa8f89c49 Jul 06 11:35:27.662448 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:27.662406 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-95wlx" event={"ID":"dfcbe27d-5c42-45f9-a719-5b43b710c11d","Type":"ContainerStarted","Data":"c2a5a24b43b0b6ec62e447fe0d0a1277bd772b800a1b3e5b1b2cf62aa8f89c49"} Jul 06 11:35:28.614120 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:28.614086 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-64c57c544d-4nq7x"] Jul 06 11:35:28.622901 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:28.622879 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64c57c544d-4nq7x" Jul 06 11:35:28.625650 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:28.625626 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 06 11:35:28.625917 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:28.625900 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-nlx55\"" Jul 06 11:35:28.626009 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:28.625992 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 11:35:28.626114 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:28.626096 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 11:35:28.627216 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:28.627165 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 06 11:35:28.629560 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:28.629539 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64c57c544d-4nq7x"] Jul 06 11:35:28.680185 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:28.680153 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7dbffb2-b3ec-4aea-a451-ef4ad10ab5f4-tmp\") pod \"kube-auth-proxy-64c57c544d-4nq7x\" (UID: \"a7dbffb2-b3ec-4aea-a451-ef4ad10ab5f4\") " pod="openshift-ingress/kube-auth-proxy-64c57c544d-4nq7x" Jul 06 11:35:28.680185 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:28.680190 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7dbffb2-b3ec-4aea-a451-ef4ad10ab5f4-tls-certs\") pod \"kube-auth-proxy-64c57c544d-4nq7x\" (UID: \"a7dbffb2-b3ec-4aea-a451-ef4ad10ab5f4\") " pod="openshift-ingress/kube-auth-proxy-64c57c544d-4nq7x" Jul 06 11:35:28.680431 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:28.680294 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms7wl\" (UniqueName: \"kubernetes.io/projected/a7dbffb2-b3ec-4aea-a451-ef4ad10ab5f4-kube-api-access-ms7wl\") pod \"kube-auth-proxy-64c57c544d-4nq7x\" (UID: \"a7dbffb2-b3ec-4aea-a451-ef4ad10ab5f4\") " pod="openshift-ingress/kube-auth-proxy-64c57c544d-4nq7x" Jul 06 11:35:28.781725 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:28.781671 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms7wl\" (UniqueName: \"kubernetes.io/projected/a7dbffb2-b3ec-4aea-a451-ef4ad10ab5f4-kube-api-access-ms7wl\") pod \"kube-auth-proxy-64c57c544d-4nq7x\" (UID: \"a7dbffb2-b3ec-4aea-a451-ef4ad10ab5f4\") " pod="openshift-ingress/kube-auth-proxy-64c57c544d-4nq7x" Jul 06 11:35:28.781923 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:28.781780 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7dbffb2-b3ec-4aea-a451-ef4ad10ab5f4-tmp\") pod \"kube-auth-proxy-64c57c544d-4nq7x\" (UID: \"a7dbffb2-b3ec-4aea-a451-ef4ad10ab5f4\") " pod="openshift-ingress/kube-auth-proxy-64c57c544d-4nq7x" Jul 06 11:35:28.781923 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:28.781810 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7dbffb2-b3ec-4aea-a451-ef4ad10ab5f4-tls-certs\") pod \"kube-auth-proxy-64c57c544d-4nq7x\" (UID: \"a7dbffb2-b3ec-4aea-a451-ef4ad10ab5f4\") " pod="openshift-ingress/kube-auth-proxy-64c57c544d-4nq7x" Jul 06 11:35:28.782082 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:35:28.781924 2568 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 06 11:35:28.782082 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:35:28.781987 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7dbffb2-b3ec-4aea-a451-ef4ad10ab5f4-tls-certs podName:a7dbffb2-b3ec-4aea-a451-ef4ad10ab5f4 nodeName:}" failed. No retries permitted until 2026-07-06 11:35:29.281968373 +0000 UTC m=+427.761767381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a7dbffb2-b3ec-4aea-a451-ef4ad10ab5f4-tls-certs") pod "kube-auth-proxy-64c57c544d-4nq7x" (UID: "a7dbffb2-b3ec-4aea-a451-ef4ad10ab5f4") : secret "kube-auth-proxy-tls" not found Jul 06 11:35:28.784639 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:28.784613 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7dbffb2-b3ec-4aea-a451-ef4ad10ab5f4-tmp\") pod \"kube-auth-proxy-64c57c544d-4nq7x\" (UID: \"a7dbffb2-b3ec-4aea-a451-ef4ad10ab5f4\") " pod="openshift-ingress/kube-auth-proxy-64c57c544d-4nq7x" Jul 06 11:35:28.793167 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:28.793143 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms7wl\" (UniqueName: \"kubernetes.io/projected/a7dbffb2-b3ec-4aea-a451-ef4ad10ab5f4-kube-api-access-ms7wl\") pod \"kube-auth-proxy-64c57c544d-4nq7x\" (UID: \"a7dbffb2-b3ec-4aea-a451-ef4ad10ab5f4\") " pod="openshift-ingress/kube-auth-proxy-64c57c544d-4nq7x" Jul 06 11:35:29.285654 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:29.285619 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7dbffb2-b3ec-4aea-a451-ef4ad10ab5f4-tls-certs\") pod \"kube-auth-proxy-64c57c544d-4nq7x\" (UID: \"a7dbffb2-b3ec-4aea-a451-ef4ad10ab5f4\") " pod="openshift-ingress/kube-auth-proxy-64c57c544d-4nq7x" Jul 06 11:35:29.288620 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:29.288593 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7dbffb2-b3ec-4aea-a451-ef4ad10ab5f4-tls-certs\") pod \"kube-auth-proxy-64c57c544d-4nq7x\" (UID: \"a7dbffb2-b3ec-4aea-a451-ef4ad10ab5f4\") " pod="openshift-ingress/kube-auth-proxy-64c57c544d-4nq7x" Jul 06 11:35:29.535602 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:29.535579 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64c57c544d-4nq7x" Jul 06 11:35:29.677419 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:29.676845 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64c57c544d-4nq7x"] Jul 06 11:35:29.679117 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:29.679043 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-kdsrp" event={"ID":"6bfe9a65-bdfc-431b-b376-40f507daf37a","Type":"ContainerStarted","Data":"e3d01f3bc1da76e336c3e5eb21b8412ae2d2bf59363f086e08d4ec1a7fc38050"} Jul 06 11:35:29.679452 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:29.679238 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-kdsrp" Jul 06 11:35:29.682030 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:35:29.682007 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7dbffb2_b3ec_4aea_a451_ef4ad10ab5f4.slice/crio-5d60b20d417a503e540d2c6b1400d01003be9f94ba2028fe4852254ba0228a0d WatchSource:0}: Error finding container 5d60b20d417a503e540d2c6b1400d01003be9f94ba2028fe4852254ba0228a0d: Status 404 returned error can't find the container with id 5d60b20d417a503e540d2c6b1400d01003be9f94ba2028fe4852254ba0228a0d Jul 06 11:35:29.701095 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:29.701044 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-kdsrp" podStartSLOduration=1.501683768 podStartE2EDuration="4.701023771s" totalImagesPullingTime="3.199340003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:35:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:35:29.698985964 +0000 UTC m=+428.178784973" watchObservedRunningTime="2026-07-06 11:35:29.701023771 +0000 UTC m=+428.180822792" Jul 06 11:35:30.685894 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:30.685860 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64c57c544d-4nq7x" event={"ID":"a7dbffb2-b3ec-4aea-a451-ef4ad10ab5f4","Type":"ContainerStarted","Data":"5d60b20d417a503e540d2c6b1400d01003be9f94ba2028fe4852254ba0228a0d"} Jul 06 11:35:31.692419 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:31.692384 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-95wlx" event={"ID":"dfcbe27d-5c42-45f9-a719-5b43b710c11d","Type":"ContainerStarted","Data":"4c57a1953e1731e7bcf16e6675cd33f180ba3139d295f794205cb22b6876a9c7"} Jul 06 11:35:31.692838 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:31.692487 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-95wlx" Jul 06 11:35:31.711738 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:31.711656 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-95wlx" podStartSLOduration=2.623674253 podStartE2EDuration="5.71163612s" totalImagesPullingTime="3.087961867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:35:31.710593805 +0000 UTC m=+430.190392826" watchObservedRunningTime="2026-07-06 11:35:31.71163612 +0000 UTC m=+430.191435137" Jul 06 11:35:32.699510 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:32.699413 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64c57c544d-4nq7x" event={"ID":"a7dbffb2-b3ec-4aea-a451-ef4ad10ab5f4","Type":"ContainerStarted","Data":"2f45fb40f167583ea81912cfd513d1bd41e7512a3a5ca922520fa5445e311aaa"} Jul 06 11:35:32.717141 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:32.717090 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-64c57c544d-4nq7x" podStartSLOduration=2.000384447 podStartE2EDuration="4.717073582s" totalImagesPullingTime="2.716689135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:35:32.716254484 +0000 UTC m=+431.196053502" watchObservedRunningTime="2026-07-06 11:35:32.717073582 +0000 UTC m=+431.196872625" Jul 06 11:35:34.305332 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:34.305297 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-56gcm"] Jul 06 11:35:34.308747 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:34.308731 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-56gcm" Jul 06 11:35:34.311755 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:34.311731 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 06 11:35:34.311755 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:34.311733 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 06 11:35:34.312011 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:34.311990 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 06 11:35:34.312076 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:34.312013 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-kmcbr\"" Jul 06 11:35:34.312121 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:34.312072 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 11:35:34.325008 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:34.324979 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-56gcm"] Jul 06 11:35:34.326634 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:34.326610 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e2c53d85-1c9b-4fba-a7b1-05cff745478b-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-56gcm\" (UID: \"e2c53d85-1c9b-4fba-a7b1-05cff745478b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-56gcm" Jul 06 11:35:34.326779 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:34.326667 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2c53d85-1c9b-4fba-a7b1-05cff745478b-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-56gcm\" (UID: \"e2c53d85-1c9b-4fba-a7b1-05cff745478b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-56gcm" Jul 06 11:35:34.326779 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:34.326723 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62hfx\" (UniqueName: \"kubernetes.io/projected/e2c53d85-1c9b-4fba-a7b1-05cff745478b-kube-api-access-62hfx\") pod \"istiod-openshift-gateway-94bb8fbfd-56gcm\" (UID: \"e2c53d85-1c9b-4fba-a7b1-05cff745478b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-56gcm" Jul 06 11:35:34.326779 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:34.326768 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2c53d85-1c9b-4fba-a7b1-05cff745478b-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-56gcm\" (UID: \"e2c53d85-1c9b-4fba-a7b1-05cff745478b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-56gcm" Jul 06 11:35:34.326898 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:34.326791 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e2c53d85-1c9b-4fba-a7b1-05cff745478b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-56gcm\" (UID: \"e2c53d85-1c9b-4fba-a7b1-05cff745478b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-56gcm" Jul 06 11:35:34.326898 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:34.326860 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e2c53d85-1c9b-4fba-a7b1-05cff745478b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-56gcm\" (UID: \"e2c53d85-1c9b-4fba-a7b1-05cff745478b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-56gcm" Jul 06 11:35:34.326980 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:34.326902 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e2c53d85-1c9b-4fba-a7b1-05cff745478b-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-56gcm\" (UID: \"e2c53d85-1c9b-4fba-a7b1-05cff745478b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-56gcm" Jul 06 11:35:34.428140 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:34.428107 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e2c53d85-1c9b-4fba-a7b1-05cff745478b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-56gcm\" (UID: \"e2c53d85-1c9b-4fba-a7b1-05cff745478b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-56gcm" Jul 06 11:35:34.428140 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:34.428151 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e2c53d85-1c9b-4fba-a7b1-05cff745478b-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-56gcm\" (UID: \"e2c53d85-1c9b-4fba-a7b1-05cff745478b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-56gcm" Jul 06 11:35:34.428412 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:34.428182 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e2c53d85-1c9b-4fba-a7b1-05cff745478b-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-56gcm\" (UID: \"e2c53d85-1c9b-4fba-a7b1-05cff745478b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-56gcm" Jul 06 11:35:34.428412 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:34.428300 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2c53d85-1c9b-4fba-a7b1-05cff745478b-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-56gcm\" (UID: \"e2c53d85-1c9b-4fba-a7b1-05cff745478b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-56gcm" Jul 06 11:35:34.428412 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:34.428344 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62hfx\" (UniqueName: \"kubernetes.io/projected/e2c53d85-1c9b-4fba-a7b1-05cff745478b-kube-api-access-62hfx\") pod \"istiod-openshift-gateway-94bb8fbfd-56gcm\" (UID: \"e2c53d85-1c9b-4fba-a7b1-05cff745478b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-56gcm" Jul 06 11:35:34.428412 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:34.428397 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2c53d85-1c9b-4fba-a7b1-05cff745478b-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-56gcm\" (UID: \"e2c53d85-1c9b-4fba-a7b1-05cff745478b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-56gcm" Jul 06 11:35:34.428610 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:34.428421 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e2c53d85-1c9b-4fba-a7b1-05cff745478b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-56gcm\" (UID: \"e2c53d85-1c9b-4fba-a7b1-05cff745478b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-56gcm" Jul 06 11:35:34.429036 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:34.429015 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e2c53d85-1c9b-4fba-a7b1-05cff745478b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-56gcm\" (UID: \"e2c53d85-1c9b-4fba-a7b1-05cff745478b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-56gcm" Jul 06 11:35:34.430576 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:34.430550 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e2c53d85-1c9b-4fba-a7b1-05cff745478b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-56gcm\" (UID: \"e2c53d85-1c9b-4fba-a7b1-05cff745478b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-56gcm" Jul 06 11:35:34.430713 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:34.430644 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e2c53d85-1c9b-4fba-a7b1-05cff745478b-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-56gcm\" (UID: \"e2c53d85-1c9b-4fba-a7b1-05cff745478b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-56gcm" Jul 06 11:35:34.430822 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:34.430804 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2c53d85-1c9b-4fba-a7b1-05cff745478b-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-56gcm\" (UID: \"e2c53d85-1c9b-4fba-a7b1-05cff745478b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-56gcm" Jul 06 11:35:34.430946 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:34.430929 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e2c53d85-1c9b-4fba-a7b1-05cff745478b-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-56gcm\" (UID: \"e2c53d85-1c9b-4fba-a7b1-05cff745478b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-56gcm" Jul 06 11:35:34.437140 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:34.437116 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2c53d85-1c9b-4fba-a7b1-05cff745478b-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-56gcm\" (UID: \"e2c53d85-1c9b-4fba-a7b1-05cff745478b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-56gcm" Jul 06 11:35:34.437262 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:34.437147 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62hfx\" (UniqueName: \"kubernetes.io/projected/e2c53d85-1c9b-4fba-a7b1-05cff745478b-kube-api-access-62hfx\") pod \"istiod-openshift-gateway-94bb8fbfd-56gcm\" (UID: \"e2c53d85-1c9b-4fba-a7b1-05cff745478b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-56gcm" Jul 06 11:35:34.618189 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:34.618099 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-56gcm" Jul 06 11:35:34.755526 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:34.755497 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-56gcm"] Jul 06 11:35:34.757474 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:35:34.757443 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c53d85_1c9b_4fba_a7b1_05cff745478b.slice/crio-e13bb2ec48d8390220d51594c5814af0d500120374afc20b12af47654d34a676 WatchSource:0}: Error finding container e13bb2ec48d8390220d51594c5814af0d500120374afc20b12af47654d34a676: Status 404 returned error can't find the container with id e13bb2ec48d8390220d51594c5814af0d500120374afc20b12af47654d34a676 Jul 06 11:35:35.534880 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.534843 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn"] Jul 06 11:35:35.541431 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.541402 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:35.545660 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.545617 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-tkf7h\"" Jul 06 11:35:35.547575 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.547550 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn"] Jul 06 11:35:35.636748 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.636710 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9859ff5-1763-4c0c-a074-3e9f86a65f68-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678n6qkn\" (UID: \"b9859ff5-1763-4c0c-a074-3e9f86a65f68\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:35.636945 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.636773 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9859ff5-1763-4c0c-a074-3e9f86a65f68-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678n6qkn\" (UID: \"b9859ff5-1763-4c0c-a074-3e9f86a65f68\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:35.636945 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.636851 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9859ff5-1763-4c0c-a074-3e9f86a65f68-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678n6qkn\" (UID: \"b9859ff5-1763-4c0c-a074-3e9f86a65f68\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:35.636945 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.636895 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9859ff5-1763-4c0c-a074-3e9f86a65f68-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678n6qkn\" (UID: \"b9859ff5-1763-4c0c-a074-3e9f86a65f68\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:35.636945 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.636919 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9859ff5-1763-4c0c-a074-3e9f86a65f68-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678n6qkn\" (UID: \"b9859ff5-1763-4c0c-a074-3e9f86a65f68\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:35.637181 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.637011 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ntvh\" (UniqueName: \"kubernetes.io/projected/b9859ff5-1763-4c0c-a074-3e9f86a65f68-kube-api-access-4ntvh\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678n6qkn\" (UID: \"b9859ff5-1763-4c0c-a074-3e9f86a65f68\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:35.637181 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.637058 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9859ff5-1763-4c0c-a074-3e9f86a65f68-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678n6qkn\" (UID: \"b9859ff5-1763-4c0c-a074-3e9f86a65f68\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:35.637181 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.637118 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9859ff5-1763-4c0c-a074-3e9f86a65f68-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678n6qkn\" (UID: \"b9859ff5-1763-4c0c-a074-3e9f86a65f68\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:35.637181 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.637157 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9859ff5-1763-4c0c-a074-3e9f86a65f68-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678n6qkn\" (UID: \"b9859ff5-1763-4c0c-a074-3e9f86a65f68\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:35.716020 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.715985 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-56gcm" event={"ID":"e2c53d85-1c9b-4fba-a7b1-05cff745478b","Type":"ContainerStarted","Data":"e13bb2ec48d8390220d51594c5814af0d500120374afc20b12af47654d34a676"} Jul 06 11:35:35.737573 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.737529 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ntvh\" (UniqueName: \"kubernetes.io/projected/b9859ff5-1763-4c0c-a074-3e9f86a65f68-kube-api-access-4ntvh\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678n6qkn\" (UID: \"b9859ff5-1763-4c0c-a074-3e9f86a65f68\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:35.737800 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.737592 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9859ff5-1763-4c0c-a074-3e9f86a65f68-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678n6qkn\" (UID: \"b9859ff5-1763-4c0c-a074-3e9f86a65f68\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:35.737800 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.737651 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9859ff5-1763-4c0c-a074-3e9f86a65f68-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678n6qkn\" (UID: \"b9859ff5-1763-4c0c-a074-3e9f86a65f68\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:35.737800 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.737714 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9859ff5-1763-4c0c-a074-3e9f86a65f68-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678n6qkn\" (UID: \"b9859ff5-1763-4c0c-a074-3e9f86a65f68\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:35.737800 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.737758 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9859ff5-1763-4c0c-a074-3e9f86a65f68-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678n6qkn\" (UID: \"b9859ff5-1763-4c0c-a074-3e9f86a65f68\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:35.737800 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.737801 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9859ff5-1763-4c0c-a074-3e9f86a65f68-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678n6qkn\" (UID: \"b9859ff5-1763-4c0c-a074-3e9f86a65f68\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:35.738077 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.737829 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9859ff5-1763-4c0c-a074-3e9f86a65f68-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678n6qkn\" (UID: \"b9859ff5-1763-4c0c-a074-3e9f86a65f68\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:35.738077 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.737854 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9859ff5-1763-4c0c-a074-3e9f86a65f68-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678n6qkn\" (UID: \"b9859ff5-1763-4c0c-a074-3e9f86a65f68\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:35.738077 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.737878 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9859ff5-1763-4c0c-a074-3e9f86a65f68-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678n6qkn\" (UID: \"b9859ff5-1763-4c0c-a074-3e9f86a65f68\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:35.738232 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.738089 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9859ff5-1763-4c0c-a074-3e9f86a65f68-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678n6qkn\" (UID: \"b9859ff5-1763-4c0c-a074-3e9f86a65f68\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:35.738232 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.738187 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9859ff5-1763-4c0c-a074-3e9f86a65f68-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678n6qkn\" (UID: \"b9859ff5-1763-4c0c-a074-3e9f86a65f68\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:35.738232 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.738204 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9859ff5-1763-4c0c-a074-3e9f86a65f68-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678n6qkn\" (UID: \"b9859ff5-1763-4c0c-a074-3e9f86a65f68\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:35.738422 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.738395 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9859ff5-1763-4c0c-a074-3e9f86a65f68-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678n6qkn\" (UID: \"b9859ff5-1763-4c0c-a074-3e9f86a65f68\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:35.738717 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.738666 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9859ff5-1763-4c0c-a074-3e9f86a65f68-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678n6qkn\" (UID: \"b9859ff5-1763-4c0c-a074-3e9f86a65f68\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:35.740505 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.740470 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9859ff5-1763-4c0c-a074-3e9f86a65f68-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678n6qkn\" (UID: \"b9859ff5-1763-4c0c-a074-3e9f86a65f68\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:35.740801 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.740772 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9859ff5-1763-4c0c-a074-3e9f86a65f68-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678n6qkn\" (UID: \"b9859ff5-1763-4c0c-a074-3e9f86a65f68\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:35.775422 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.775390 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9859ff5-1763-4c0c-a074-3e9f86a65f68-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678n6qkn\" (UID: \"b9859ff5-1763-4c0c-a074-3e9f86a65f68\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:35.775692 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.775653 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ntvh\" (UniqueName: \"kubernetes.io/projected/b9859ff5-1763-4c0c-a074-3e9f86a65f68-kube-api-access-4ntvh\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678n6qkn\" (UID: \"b9859ff5-1763-4c0c-a074-3e9f86a65f68\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:35.859341 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:35.859250 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:36.059741 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:36.059712 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn"] Jul 06 11:35:36.060695 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:35:36.060653 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9859ff5_1763_4c0c_a074_3e9f86a65f68.slice/crio-ec1acf9209b6149a4526bccf14ad7882dd43d0731cb48cf003b421e278618a4a WatchSource:0}: Error finding container ec1acf9209b6149a4526bccf14ad7882dd43d0731cb48cf003b421e278618a4a: Status 404 returned error can't find the container with id ec1acf9209b6149a4526bccf14ad7882dd43d0731cb48cf003b421e278618a4a Jul 06 11:35:36.723246 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:36.723210 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" event={"ID":"b9859ff5-1763-4c0c-a074-3e9f86a65f68","Type":"ContainerStarted","Data":"ec1acf9209b6149a4526bccf14ad7882dd43d0731cb48cf003b421e278618a4a"} Jul 06 11:35:37.272538 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:37.272500 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 11:35:37.272619 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:37.272572 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 11:35:37.741146 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:37.740850 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-56gcm" event={"ID":"e2c53d85-1c9b-4fba-a7b1-05cff745478b","Type":"ContainerStarted","Data":"e9699061f6146216666bc4b832b0cd2ec8a4a6ba1621bac692ca95d78b0b8337"} Jul 06 11:35:37.741146 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:37.740921 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-56gcm" Jul 06 11:35:37.775294 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:37.775234 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-56gcm" podStartSLOduration=1.26295873 podStartE2EDuration="3.77520922s" totalImagesPullingTime="2.51225049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:35:37.772345478 +0000 UTC m=+436.252144495" watchObservedRunningTime="2026-07-06 11:35:37.77520922 +0000 UTC m=+436.255008239" Jul 06 11:35:37.852637 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:37.852519 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gh8vj"] Jul 06 11:35:37.856252 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:37.856220 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gh8vj" Jul 06 11:35:37.860526 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:37.860114 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-6w6fk\"" Jul 06 11:35:37.860526 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:37.860370 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 06 11:35:37.871915 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:37.871886 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-6zprf"] Jul 06 11:35:37.875440 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:37.875417 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gh8vj"] Jul 06 11:35:37.875576 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:37.875523 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6zprf" Jul 06 11:35:37.879628 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:37.879603 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 06 11:35:37.879898 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:37.879870 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-dcglv\"" Jul 06 11:35:37.907534 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:37.907463 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-6zprf"] Jul 06 11:35:37.962140 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:37.962074 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kdg9\" (UniqueName: \"kubernetes.io/projected/4db87dba-c23e-4a77-b58b-b9de0644d486-kube-api-access-4kdg9\") pod \"llmisvc-controller-manager-777c6bd7d5-6zprf\" (UID: \"4db87dba-c23e-4a77-b58b-b9de0644d486\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6zprf" Jul 06 11:35:37.962336 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:37.962197 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/609ac905-0773-4403-8e9b-8fe8a4dcf908-cert\") pod \"kserve-controller-manager-d6f4b65b8-gh8vj\" (UID: \"609ac905-0773-4403-8e9b-8fe8a4dcf908\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gh8vj" Jul 06 11:35:37.962336 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:37.962235 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4db87dba-c23e-4a77-b58b-b9de0644d486-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6zprf\" (UID: \"4db87dba-c23e-4a77-b58b-b9de0644d486\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6zprf" Jul 06 11:35:37.962336 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:37.962266 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg9dd\" (UniqueName: \"kubernetes.io/projected/609ac905-0773-4403-8e9b-8fe8a4dcf908-kube-api-access-wg9dd\") pod \"kserve-controller-manager-d6f4b65b8-gh8vj\" (UID: \"609ac905-0773-4403-8e9b-8fe8a4dcf908\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gh8vj" Jul 06 11:35:38.063323 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:38.063228 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kdg9\" (UniqueName: \"kubernetes.io/projected/4db87dba-c23e-4a77-b58b-b9de0644d486-kube-api-access-4kdg9\") pod \"llmisvc-controller-manager-777c6bd7d5-6zprf\" (UID: \"4db87dba-c23e-4a77-b58b-b9de0644d486\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6zprf" Jul 06 11:35:38.063323 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:38.063285 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/609ac905-0773-4403-8e9b-8fe8a4dcf908-cert\") pod \"kserve-controller-manager-d6f4b65b8-gh8vj\" (UID: \"609ac905-0773-4403-8e9b-8fe8a4dcf908\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gh8vj" Jul 06 11:35:38.063563 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:38.063325 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4db87dba-c23e-4a77-b58b-b9de0644d486-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6zprf\" (UID: \"4db87dba-c23e-4a77-b58b-b9de0644d486\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6zprf" Jul 06 11:35:38.063563 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:38.063350 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg9dd\" (UniqueName: \"kubernetes.io/projected/609ac905-0773-4403-8e9b-8fe8a4dcf908-kube-api-access-wg9dd\") pod \"kserve-controller-manager-d6f4b65b8-gh8vj\" (UID: \"609ac905-0773-4403-8e9b-8fe8a4dcf908\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gh8vj" Jul 06 11:35:38.063563 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:35:38.063471 2568 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 06 11:35:38.063563 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:35:38.063558 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/609ac905-0773-4403-8e9b-8fe8a4dcf908-cert podName:609ac905-0773-4403-8e9b-8fe8a4dcf908 nodeName:}" failed. No retries permitted until 2026-07-06 11:35:38.563534748 +0000 UTC m=+437.043333762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/609ac905-0773-4403-8e9b-8fe8a4dcf908-cert") pod "kserve-controller-manager-d6f4b65b8-gh8vj" (UID: "609ac905-0773-4403-8e9b-8fe8a4dcf908") : secret "kserve-webhook-server-cert" not found Jul 06 11:35:38.066299 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:38.066273 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4db87dba-c23e-4a77-b58b-b9de0644d486-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6zprf\" (UID: \"4db87dba-c23e-4a77-b58b-b9de0644d486\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6zprf" Jul 06 11:35:38.077885 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:38.077861 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg9dd\" (UniqueName: \"kubernetes.io/projected/609ac905-0773-4403-8e9b-8fe8a4dcf908-kube-api-access-wg9dd\") pod \"kserve-controller-manager-d6f4b65b8-gh8vj\" (UID: \"609ac905-0773-4403-8e9b-8fe8a4dcf908\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gh8vj" Jul 06 11:35:38.083096 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:38.083032 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kdg9\" (UniqueName: \"kubernetes.io/projected/4db87dba-c23e-4a77-b58b-b9de0644d486-kube-api-access-4kdg9\") pod \"llmisvc-controller-manager-777c6bd7d5-6zprf\" (UID: \"4db87dba-c23e-4a77-b58b-b9de0644d486\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6zprf" Jul 06 11:35:38.188308 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:38.188242 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6zprf" Jul 06 11:35:38.347910 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:38.347872 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-6zprf"] Jul 06 11:35:38.349256 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:35:38.349227 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4db87dba_c23e_4a77_b58b_b9de0644d486.slice/crio-6b324de8dcb0f63c1b7699aa2a8c54fb43b24b6bd365b4b60bbbd6e0964effd3 WatchSource:0}: Error finding container 6b324de8dcb0f63c1b7699aa2a8c54fb43b24b6bd365b4b60bbbd6e0964effd3: Status 404 returned error can't find the container with id 6b324de8dcb0f63c1b7699aa2a8c54fb43b24b6bd365b4b60bbbd6e0964effd3 Jul 06 11:35:38.568949 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:38.568908 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/609ac905-0773-4403-8e9b-8fe8a4dcf908-cert\") pod \"kserve-controller-manager-d6f4b65b8-gh8vj\" (UID: \"609ac905-0773-4403-8e9b-8fe8a4dcf908\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gh8vj" Jul 06 11:35:38.569117 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:35:38.569091 2568 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 06 11:35:38.569195 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:35:38.569174 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/609ac905-0773-4403-8e9b-8fe8a4dcf908-cert podName:609ac905-0773-4403-8e9b-8fe8a4dcf908 nodeName:}" failed. No retries permitted until 2026-07-06 11:35:39.569153497 +0000 UTC m=+438.048952495 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/609ac905-0773-4403-8e9b-8fe8a4dcf908-cert") pod "kserve-controller-manager-d6f4b65b8-gh8vj" (UID: "609ac905-0773-4403-8e9b-8fe8a4dcf908") : secret "kserve-webhook-server-cert" not found Jul 06 11:35:38.747077 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:38.747040 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6zprf" event={"ID":"4db87dba-c23e-4a77-b58b-b9de0644d486","Type":"ContainerStarted","Data":"6b324de8dcb0f63c1b7699aa2a8c54fb43b24b6bd365b4b60bbbd6e0964effd3"} Jul 06 11:35:38.749143 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:38.749115 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-56gcm" Jul 06 11:35:39.505773 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:39.505732 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 11:35:39.505876 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:39.505806 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 11:35:39.505876 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:39.505835 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 11:35:39.580394 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:39.580354 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/609ac905-0773-4403-8e9b-8fe8a4dcf908-cert\") pod \"kserve-controller-manager-d6f4b65b8-gh8vj\" (UID: \"609ac905-0773-4403-8e9b-8fe8a4dcf908\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gh8vj" Jul 06 11:35:39.583185 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:39.583148 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/609ac905-0773-4403-8e9b-8fe8a4dcf908-cert\") pod \"kserve-controller-manager-d6f4b65b8-gh8vj\" (UID: \"609ac905-0773-4403-8e9b-8fe8a4dcf908\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gh8vj" Jul 06 11:35:39.674885 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:39.674854 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gh8vj" Jul 06 11:35:39.755514 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:39.755447 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" event={"ID":"b9859ff5-1763-4c0c-a074-3e9f86a65f68","Type":"ContainerStarted","Data":"5ce4d0d373b13d9e239dd706ab15b0a485e41b30f94a6974bc2775a7ad326c13"} Jul 06 11:35:39.859075 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:39.859019 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" podStartSLOduration=1.416951464 podStartE2EDuration="4.858995003s" totalImagesPullingTime="3.442043539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:35:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:35:39.778748579 +0000 UTC m=+438.258547597" watchObservedRunningTime="2026-07-06 11:35:39.858995003 +0000 UTC m=+438.338794019" Jul 06 11:35:39.860024 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:39.859945 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:39.861502 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:39.861471 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gh8vj"] Jul 06 11:35:39.916987 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:35:39.916947 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod609ac905_0773_4403_8e9b_8fe8a4dcf908.slice/crio-4e37cb239b2b22829cdc47a82a32f59f7296c884777c1c305a92396331fa7be0 WatchSource:0}: Error finding container 4e37cb239b2b22829cdc47a82a32f59f7296c884777c1c305a92396331fa7be0: Status 404 returned error can't find the container with id 4e37cb239b2b22829cdc47a82a32f59f7296c884777c1c305a92396331fa7be0 Jul 06 11:35:40.688323 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:40.688293 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-kdsrp" Jul 06 11:35:40.761582 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:40.761541 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gh8vj" event={"ID":"609ac905-0773-4403-8e9b-8fe8a4dcf908","Type":"ContainerStarted","Data":"4e37cb239b2b22829cdc47a82a32f59f7296c884777c1c305a92396331fa7be0"} Jul 06 11:35:40.865786 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:40.865749 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:41.768846 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:41.768810 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:41.769969 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:41.769932 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678n6qkn" Jul 06 11:35:42.704473 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:42.704442 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-95wlx" Jul 06 11:35:43.780503 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:43.780463 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6zprf" event={"ID":"4db87dba-c23e-4a77-b58b-b9de0644d486","Type":"ContainerStarted","Data":"68346f720677d4b7f437d394ea65542eb0877eb0e149083dedcc4603f45ea626"} Jul 06 11:35:43.781024 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:43.780582 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6zprf" Jul 06 11:35:43.782039 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:43.782016 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gh8vj" event={"ID":"609ac905-0773-4403-8e9b-8fe8a4dcf908","Type":"ContainerStarted","Data":"0df1e9dbaffa20d13fdd60c5c8c1cd4698891c1236161cd22d761f3f6758303f"} Jul 06 11:35:43.782200 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:43.782186 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gh8vj" Jul 06 11:35:43.801259 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:43.801213 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6zprf" podStartSLOduration=2.358087812 podStartE2EDuration="6.801189451s" totalImagesPullingTime="4.443101639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:35:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:35:43.799911123 +0000 UTC m=+442.279710140" watchObservedRunningTime="2026-07-06 11:35:43.801189451 +0000 UTC m=+442.280988461" Jul 06 11:35:43.818740 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:35:43.818692 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gh8vj" podStartSLOduration=3.9382977070000003 podStartE2EDuration="6.818659736s" totalImagesPullingTime="2.880362029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:35:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:35:43.817795158 +0000 UTC m=+442.297594177" watchObservedRunningTime="2026-07-06 11:35:43.818659736 +0000 UTC m=+442.298458776" Jul 06 11:36:14.788442 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:14.788411 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6zprf" Jul 06 11:36:14.791403 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:14.791381 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gh8vj" Jul 06 11:36:42.108951 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:42.108918 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb"] Jul 06 11:36:42.116622 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:42.116595 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb" Jul 06 11:36:42.119973 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:42.119949 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 11:36:42.121285 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:42.121264 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 11:36:42.121410 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:42.121297 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-sk7l8\"" Jul 06 11:36:42.124083 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:42.124059 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb"] Jul 06 11:36:42.228897 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:42.228864 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6085760-c8a0-4f8c-9494-e86fd58675ac-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb\" (UID: \"a6085760-c8a0-4f8c-9494-e86fd58675ac\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb" Jul 06 11:36:42.229090 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:42.228914 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6085760-c8a0-4f8c-9494-e86fd58675ac-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb\" (UID: \"a6085760-c8a0-4f8c-9494-e86fd58675ac\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb" Jul 06 11:36:42.229090 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:42.229011 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk6m8\" (UniqueName: \"kubernetes.io/projected/a6085760-c8a0-4f8c-9494-e86fd58675ac-kube-api-access-gk6m8\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb\" (UID: \"a6085760-c8a0-4f8c-9494-e86fd58675ac\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb" Jul 06 11:36:42.330323 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:42.330286 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6085760-c8a0-4f8c-9494-e86fd58675ac-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb\" (UID: \"a6085760-c8a0-4f8c-9494-e86fd58675ac\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb" Jul 06 11:36:42.330482 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:42.330350 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk6m8\" (UniqueName: \"kubernetes.io/projected/a6085760-c8a0-4f8c-9494-e86fd58675ac-kube-api-access-gk6m8\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb\" (UID: \"a6085760-c8a0-4f8c-9494-e86fd58675ac\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb" Jul 06 11:36:42.330482 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:42.330386 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6085760-c8a0-4f8c-9494-e86fd58675ac-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb\" (UID: \"a6085760-c8a0-4f8c-9494-e86fd58675ac\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb" Jul 06 11:36:42.330717 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:42.330670 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6085760-c8a0-4f8c-9494-e86fd58675ac-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb\" (UID: \"a6085760-c8a0-4f8c-9494-e86fd58675ac\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb" Jul 06 11:36:42.330760 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:42.330717 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6085760-c8a0-4f8c-9494-e86fd58675ac-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb\" (UID: \"a6085760-c8a0-4f8c-9494-e86fd58675ac\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb" Jul 06 11:36:42.339794 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:42.339771 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk6m8\" (UniqueName: \"kubernetes.io/projected/a6085760-c8a0-4f8c-9494-e86fd58675ac-kube-api-access-gk6m8\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb\" (UID: \"a6085760-c8a0-4f8c-9494-e86fd58675ac\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb" Jul 06 11:36:42.427998 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:42.427975 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb" Jul 06 11:36:42.557165 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:42.557137 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb"] Jul 06 11:36:42.559011 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:36:42.558983 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6085760_c8a0_4f8c_9494_e86fd58675ac.slice/crio-899ca5ca8b367acfaa31968838acfdc3f24fd2a3f60448250aa71f0f4f3d0ae6 WatchSource:0}: Error finding container 899ca5ca8b367acfaa31968838acfdc3f24fd2a3f60448250aa71f0f4f3d0ae6: Status 404 returned error can't find the container with id 899ca5ca8b367acfaa31968838acfdc3f24fd2a3f60448250aa71f0f4f3d0ae6 Jul 06 11:36:42.713987 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:42.713926 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql"] Jul 06 11:36:42.716808 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:42.716794 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql" Jul 06 11:36:42.732731 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:42.732704 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql"] Jul 06 11:36:42.833935 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:42.833900 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prxtv\" (UniqueName: \"kubernetes.io/projected/eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540-kube-api-access-prxtv\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql\" (UID: \"eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql" Jul 06 11:36:42.834100 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:42.833969 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql\" (UID: \"eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql" Jul 06 11:36:42.834100 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:42.834027 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql\" (UID: \"eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql" Jul 06 11:36:42.934998 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:42.934955 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql\" (UID: \"eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql" Jul 06 11:36:42.934998 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:42.935003 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql\" (UID: \"eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql" Jul 06 11:36:42.935194 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:42.935034 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prxtv\" (UniqueName: \"kubernetes.io/projected/eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540-kube-api-access-prxtv\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql\" (UID: \"eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql" Jul 06 11:36:42.935329 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:42.935310 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql\" (UID: \"eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql" Jul 06 11:36:42.935369 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:42.935347 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql\" (UID: \"eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql" Jul 06 11:36:42.948445 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:42.948421 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prxtv\" (UniqueName: \"kubernetes.io/projected/eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540-kube-api-access-prxtv\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql\" (UID: \"eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql" Jul 06 11:36:43.038414 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.038329 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql" Jul 06 11:36:43.051004 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.050968 2568 generic.go:356] "Generic (PLEG): container finished" podID="a6085760-c8a0-4f8c-9494-e86fd58675ac" containerID="219af9b8c7f7164ee10731f9c785adf097908468156b67f340078b677b4901c5" exitCode=0 Jul 06 11:36:43.051137 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.051061 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb" event={"ID":"a6085760-c8a0-4f8c-9494-e86fd58675ac","Type":"ContainerDied","Data":"219af9b8c7f7164ee10731f9c785adf097908468156b67f340078b677b4901c5"} Jul 06 11:36:43.051137 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.051122 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb" event={"ID":"a6085760-c8a0-4f8c-9494-e86fd58675ac","Type":"ContainerStarted","Data":"899ca5ca8b367acfaa31968838acfdc3f24fd2a3f60448250aa71f0f4f3d0ae6"} Jul 06 11:36:43.117038 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.117005 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf"] Jul 06 11:36:43.120946 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.120918 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf" Jul 06 11:36:43.131770 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.131748 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf"] Jul 06 11:36:43.171488 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.171440 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql"] Jul 06 11:36:43.173621 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:36:43.173590 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda6f93f_8f1d_49ba_a3dd_7e1cc64f1540.slice/crio-25a6e4c3ce1d6dab7977903210da745d0b338e36982928ab73d2f868be396e9f WatchSource:0}: Error finding container 25a6e4c3ce1d6dab7977903210da745d0b338e36982928ab73d2f868be396e9f: Status 404 returned error can't find the container with id 25a6e4c3ce1d6dab7977903210da745d0b338e36982928ab73d2f868be396e9f Jul 06 11:36:43.237942 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.237918 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f247d94-5720-4a5e-af09-61639affce37-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf\" (UID: \"6f247d94-5720-4a5e-af09-61639affce37\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf" Jul 06 11:36:43.238091 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.237988 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f247d94-5720-4a5e-af09-61639affce37-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf\" (UID: \"6f247d94-5720-4a5e-af09-61639affce37\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf" Jul 06 11:36:43.238091 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.238040 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmk2g\" (UniqueName: \"kubernetes.io/projected/6f247d94-5720-4a5e-af09-61639affce37-kube-api-access-kmk2g\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf\" (UID: \"6f247d94-5720-4a5e-af09-61639affce37\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf" Jul 06 11:36:43.339319 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.339210 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f247d94-5720-4a5e-af09-61639affce37-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf\" (UID: \"6f247d94-5720-4a5e-af09-61639affce37\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf" Jul 06 11:36:43.339319 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.339279 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f247d94-5720-4a5e-af09-61639affce37-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf\" (UID: \"6f247d94-5720-4a5e-af09-61639affce37\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf" Jul 06 11:36:43.339319 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.339314 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmk2g\" (UniqueName: \"kubernetes.io/projected/6f247d94-5720-4a5e-af09-61639affce37-kube-api-access-kmk2g\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf\" (UID: \"6f247d94-5720-4a5e-af09-61639affce37\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf" Jul 06 11:36:43.339586 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.339572 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f247d94-5720-4a5e-af09-61639affce37-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf\" (UID: \"6f247d94-5720-4a5e-af09-61639affce37\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf" Jul 06 11:36:43.339659 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.339637 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f247d94-5720-4a5e-af09-61639affce37-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf\" (UID: \"6f247d94-5720-4a5e-af09-61639affce37\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf" Jul 06 11:36:43.349081 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.349058 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmk2g\" (UniqueName: \"kubernetes.io/projected/6f247d94-5720-4a5e-af09-61639affce37-kube-api-access-kmk2g\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf\" (UID: \"6f247d94-5720-4a5e-af09-61639affce37\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf" Jul 06 11:36:43.433468 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.433436 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf" Jul 06 11:36:43.509949 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.509919 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx"] Jul 06 11:36:43.513346 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.513323 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx" Jul 06 11:36:43.524243 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.524219 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx"] Jul 06 11:36:43.568803 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.568717 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf"] Jul 06 11:36:43.570864 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:36:43.570835 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f247d94_5720_4a5e_af09_61639affce37.slice/crio-b14c9d29438555a0ea814fd04b409d600ab9e354f3f61f2e551f762709249e69 WatchSource:0}: Error finding container b14c9d29438555a0ea814fd04b409d600ab9e354f3f61f2e551f762709249e69: Status 404 returned error can't find the container with id b14c9d29438555a0ea814fd04b409d600ab9e354f3f61f2e551f762709249e69 Jul 06 11:36:43.642896 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.642866 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/969a11c9-a0fa-45d1-b093-4bb6c5fc84b4-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx\" (UID: \"969a11c9-a0fa-45d1-b093-4bb6c5fc84b4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx" Jul 06 11:36:43.643002 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.642911 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/969a11c9-a0fa-45d1-b093-4bb6c5fc84b4-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx\" (UID: \"969a11c9-a0fa-45d1-b093-4bb6c5fc84b4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx" Jul 06 11:36:43.643002 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.642953 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtblr\" (UniqueName: \"kubernetes.io/projected/969a11c9-a0fa-45d1-b093-4bb6c5fc84b4-kube-api-access-qtblr\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx\" (UID: \"969a11c9-a0fa-45d1-b093-4bb6c5fc84b4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx" Jul 06 11:36:43.744418 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.744388 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/969a11c9-a0fa-45d1-b093-4bb6c5fc84b4-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx\" (UID: \"969a11c9-a0fa-45d1-b093-4bb6c5fc84b4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx" Jul 06 11:36:43.744583 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.744426 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/969a11c9-a0fa-45d1-b093-4bb6c5fc84b4-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx\" (UID: \"969a11c9-a0fa-45d1-b093-4bb6c5fc84b4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx" Jul 06 11:36:43.744583 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.744471 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtblr\" (UniqueName: \"kubernetes.io/projected/969a11c9-a0fa-45d1-b093-4bb6c5fc84b4-kube-api-access-qtblr\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx\" (UID: \"969a11c9-a0fa-45d1-b093-4bb6c5fc84b4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx" Jul 06 11:36:43.744863 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.744837 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/969a11c9-a0fa-45d1-b093-4bb6c5fc84b4-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx\" (UID: \"969a11c9-a0fa-45d1-b093-4bb6c5fc84b4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx" Jul 06 11:36:43.744901 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.744880 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/969a11c9-a0fa-45d1-b093-4bb6c5fc84b4-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx\" (UID: \"969a11c9-a0fa-45d1-b093-4bb6c5fc84b4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx" Jul 06 11:36:43.754447 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.754421 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtblr\" (UniqueName: \"kubernetes.io/projected/969a11c9-a0fa-45d1-b093-4bb6c5fc84b4-kube-api-access-qtblr\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx\" (UID: \"969a11c9-a0fa-45d1-b093-4bb6c5fc84b4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx" Jul 06 11:36:43.824978 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.824945 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx" Jul 06 11:36:43.996823 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:43.996793 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx"] Jul 06 11:36:43.997637 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:36:43.997612 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod969a11c9_a0fa_45d1_b093_4bb6c5fc84b4.slice/crio-1ea0e22a9dd79a957daaf43eef1fb566006ab21eaea420efa224e7db5b9c9087 WatchSource:0}: Error finding container 1ea0e22a9dd79a957daaf43eef1fb566006ab21eaea420efa224e7db5b9c9087: Status 404 returned error can't find the container with id 1ea0e22a9dd79a957daaf43eef1fb566006ab21eaea420efa224e7db5b9c9087 Jul 06 11:36:44.056341 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:44.056312 2568 generic.go:356] "Generic (PLEG): container finished" podID="eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540" containerID="f3f5bbb52bcd07a709550fbd35786e1920374dd2f3c2f1c6dfd7cf4b84af5870" exitCode=0 Jul 06 11:36:44.056479 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:44.056385 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql" event={"ID":"eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540","Type":"ContainerDied","Data":"f3f5bbb52bcd07a709550fbd35786e1920374dd2f3c2f1c6dfd7cf4b84af5870"} Jul 06 11:36:44.056479 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:44.056428 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql" event={"ID":"eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540","Type":"ContainerStarted","Data":"25a6e4c3ce1d6dab7977903210da745d0b338e36982928ab73d2f868be396e9f"} Jul 06 11:36:44.057527 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:44.057506 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx" event={"ID":"969a11c9-a0fa-45d1-b093-4bb6c5fc84b4","Type":"ContainerStarted","Data":"1ea0e22a9dd79a957daaf43eef1fb566006ab21eaea420efa224e7db5b9c9087"} Jul 06 11:36:44.058852 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:44.058834 2568 generic.go:356] "Generic (PLEG): container finished" podID="6f247d94-5720-4a5e-af09-61639affce37" containerID="1bd73e573f725ec5a71a6694f92d0a6851511bdf9c71250544f22204bdbf7f82" exitCode=0 Jul 06 11:36:44.058938 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:44.058910 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf" event={"ID":"6f247d94-5720-4a5e-af09-61639affce37","Type":"ContainerDied","Data":"1bd73e573f725ec5a71a6694f92d0a6851511bdf9c71250544f22204bdbf7f82"} Jul 06 11:36:44.058983 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:44.058942 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf" event={"ID":"6f247d94-5720-4a5e-af09-61639affce37","Type":"ContainerStarted","Data":"b14c9d29438555a0ea814fd04b409d600ab9e354f3f61f2e551f762709249e69"} Jul 06 11:36:44.060616 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:44.060592 2568 generic.go:356] "Generic (PLEG): container finished" podID="a6085760-c8a0-4f8c-9494-e86fd58675ac" containerID="40db7befc552ac9de52e96d22fe2712988bf4279ee027f2ac16d5b9af59670d7" exitCode=0 Jul 06 11:36:44.060747 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:44.060630 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb" event={"ID":"a6085760-c8a0-4f8c-9494-e86fd58675ac","Type":"ContainerDied","Data":"40db7befc552ac9de52e96d22fe2712988bf4279ee027f2ac16d5b9af59670d7"} Jul 06 11:36:45.068111 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:45.068008 2568 generic.go:356] "Generic (PLEG): container finished" podID="969a11c9-a0fa-45d1-b093-4bb6c5fc84b4" containerID="85db162e0ef90678a898c75064e759ba917905f53feb137e60bf2b0f14f71f18" exitCode=0 Jul 06 11:36:45.068535 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:45.068097 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx" event={"ID":"969a11c9-a0fa-45d1-b093-4bb6c5fc84b4","Type":"ContainerDied","Data":"85db162e0ef90678a898c75064e759ba917905f53feb137e60bf2b0f14f71f18"} Jul 06 11:36:45.070084 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:45.070058 2568 generic.go:356] "Generic (PLEG): container finished" podID="6f247d94-5720-4a5e-af09-61639affce37" containerID="ae0a91cc8acdafec8f7ee3421751db133561d9fc0f56548033f8f6eeaef714ec" exitCode=0 Jul 06 11:36:45.070206 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:45.070137 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf" event={"ID":"6f247d94-5720-4a5e-af09-61639affce37","Type":"ContainerDied","Data":"ae0a91cc8acdafec8f7ee3421751db133561d9fc0f56548033f8f6eeaef714ec"} Jul 06 11:36:45.072493 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:45.072471 2568 generic.go:356] "Generic (PLEG): container finished" podID="a6085760-c8a0-4f8c-9494-e86fd58675ac" containerID="093162572c4660aa56833c6d13a274b9e154a6b1a082e352557da251ebfb2808" exitCode=0 Jul 06 11:36:45.072574 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:45.072501 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb" event={"ID":"a6085760-c8a0-4f8c-9494-e86fd58675ac","Type":"ContainerDied","Data":"093162572c4660aa56833c6d13a274b9e154a6b1a082e352557da251ebfb2808"} Jul 06 11:36:45.074383 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:45.074364 2568 generic.go:356] "Generic (PLEG): container finished" podID="eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540" containerID="e871a8a96c142da4ffc6a3f6e28011b7385851df89b68d040aa186f41afd9018" exitCode=0 Jul 06 11:36:45.074449 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:45.074400 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql" event={"ID":"eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540","Type":"ContainerDied","Data":"e871a8a96c142da4ffc6a3f6e28011b7385851df89b68d040aa186f41afd9018"} Jul 06 11:36:46.080518 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:46.080482 2568 generic.go:356] "Generic (PLEG): container finished" podID="eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540" containerID="d3b133804ebed488f9d681aa87af44327d2aa22eb18b14787962ee1e91b49bde" exitCode=0 Jul 06 11:36:46.081000 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:46.080554 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql" event={"ID":"eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540","Type":"ContainerDied","Data":"d3b133804ebed488f9d681aa87af44327d2aa22eb18b14787962ee1e91b49bde"} Jul 06 11:36:46.082355 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:46.082332 2568 generic.go:356] "Generic (PLEG): container finished" podID="969a11c9-a0fa-45d1-b093-4bb6c5fc84b4" containerID="5111ddd5dea85a91936d539d44bc0ba33992f06c8f052983fa5145b70aa8d555" exitCode=0 Jul 06 11:36:46.082485 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:46.082418 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx" event={"ID":"969a11c9-a0fa-45d1-b093-4bb6c5fc84b4","Type":"ContainerDied","Data":"5111ddd5dea85a91936d539d44bc0ba33992f06c8f052983fa5145b70aa8d555"} Jul 06 11:36:46.084525 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:46.084502 2568 generic.go:356] "Generic (PLEG): container finished" podID="6f247d94-5720-4a5e-af09-61639affce37" containerID="082491b84a893f50d3630eb561e8e9b8c95f5bfd8986e1efc66f80555431b30d" exitCode=0 Jul 06 11:36:46.084619 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:46.084598 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf" event={"ID":"6f247d94-5720-4a5e-af09-61639affce37","Type":"ContainerDied","Data":"082491b84a893f50d3630eb561e8e9b8c95f5bfd8986e1efc66f80555431b30d"} Jul 06 11:36:46.211862 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:46.211839 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb" Jul 06 11:36:46.370560 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:46.370484 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gk6m8\" (UniqueName: \"kubernetes.io/projected/a6085760-c8a0-4f8c-9494-e86fd58675ac-kube-api-access-gk6m8\") pod \"a6085760-c8a0-4f8c-9494-e86fd58675ac\" (UID: \"a6085760-c8a0-4f8c-9494-e86fd58675ac\") " Jul 06 11:36:46.370560 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:46.370532 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6085760-c8a0-4f8c-9494-e86fd58675ac-util\") pod \"a6085760-c8a0-4f8c-9494-e86fd58675ac\" (UID: \"a6085760-c8a0-4f8c-9494-e86fd58675ac\") " Jul 06 11:36:46.370763 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:46.370565 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6085760-c8a0-4f8c-9494-e86fd58675ac-bundle\") pod \"a6085760-c8a0-4f8c-9494-e86fd58675ac\" (UID: \"a6085760-c8a0-4f8c-9494-e86fd58675ac\") " Jul 06 11:36:46.371188 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:46.371163 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6085760-c8a0-4f8c-9494-e86fd58675ac-bundle" (OuterVolumeSpecName: "bundle") pod "a6085760-c8a0-4f8c-9494-e86fd58675ac" (UID: "a6085760-c8a0-4f8c-9494-e86fd58675ac"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:36:46.372801 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:46.372779 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6085760-c8a0-4f8c-9494-e86fd58675ac-kube-api-access-gk6m8" (OuterVolumeSpecName: "kube-api-access-gk6m8") pod "a6085760-c8a0-4f8c-9494-e86fd58675ac" (UID: "a6085760-c8a0-4f8c-9494-e86fd58675ac"). InnerVolumeSpecName "kube-api-access-gk6m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 11:36:46.378655 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:46.378626 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6085760-c8a0-4f8c-9494-e86fd58675ac-util" (OuterVolumeSpecName: "util") pod "a6085760-c8a0-4f8c-9494-e86fd58675ac" (UID: "a6085760-c8a0-4f8c-9494-e86fd58675ac"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:36:46.471411 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:46.471379 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk6m8\" (UniqueName: \"kubernetes.io/projected/a6085760-c8a0-4f8c-9494-e86fd58675ac-kube-api-access-gk6m8\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:36:46.471411 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:46.471410 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a6085760-c8a0-4f8c-9494-e86fd58675ac-util\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:36:46.471617 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:46.471421 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a6085760-c8a0-4f8c-9494-e86fd58675ac-bundle\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:36:47.090918 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:47.090882 2568 generic.go:356] "Generic (PLEG): container finished" podID="969a11c9-a0fa-45d1-b093-4bb6c5fc84b4" containerID="d9966f032f99f292337a7315d0a349c2685d3cca63bb8a5f7b7ffb55deb4f66f" exitCode=0 Jul 06 11:36:47.091330 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:47.090958 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx" event={"ID":"969a11c9-a0fa-45d1-b093-4bb6c5fc84b4","Type":"ContainerDied","Data":"d9966f032f99f292337a7315d0a349c2685d3cca63bb8a5f7b7ffb55deb4f66f"} Jul 06 11:36:47.092746 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:47.092725 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb" Jul 06 11:36:47.092746 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:47.092729 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb" event={"ID":"a6085760-c8a0-4f8c-9494-e86fd58675ac","Type":"ContainerDied","Data":"899ca5ca8b367acfaa31968838acfdc3f24fd2a3f60448250aa71f0f4f3d0ae6"} Jul 06 11:36:47.092921 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:47.092782 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="899ca5ca8b367acfaa31968838acfdc3f24fd2a3f60448250aa71f0f4f3d0ae6" Jul 06 11:36:47.226864 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:47.226841 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql" Jul 06 11:36:47.239917 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:47.239899 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf" Jul 06 11:36:47.379906 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:47.379827 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmk2g\" (UniqueName: \"kubernetes.io/projected/6f247d94-5720-4a5e-af09-61639affce37-kube-api-access-kmk2g\") pod \"6f247d94-5720-4a5e-af09-61639affce37\" (UID: \"6f247d94-5720-4a5e-af09-61639affce37\") " Jul 06 11:36:47.379906 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:47.379884 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prxtv\" (UniqueName: \"kubernetes.io/projected/eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540-kube-api-access-prxtv\") pod \"eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540\" (UID: \"eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540\") " Jul 06 11:36:47.380407 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:47.379920 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f247d94-5720-4a5e-af09-61639affce37-bundle\") pod \"6f247d94-5720-4a5e-af09-61639affce37\" (UID: \"6f247d94-5720-4a5e-af09-61639affce37\") " Jul 06 11:36:47.380407 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:47.379974 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540-bundle\") pod \"eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540\" (UID: \"eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540\") " Jul 06 11:36:47.380407 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:47.380000 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f247d94-5720-4a5e-af09-61639affce37-util\") pod \"6f247d94-5720-4a5e-af09-61639affce37\" (UID: \"6f247d94-5720-4a5e-af09-61639affce37\") " Jul 06 11:36:47.380407 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:47.380037 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540-util\") pod \"eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540\" (UID: \"eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540\") " Jul 06 11:36:47.380628 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:47.380490 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540-bundle" (OuterVolumeSpecName: "bundle") pod "eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540" (UID: "eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:36:47.381121 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:47.381097 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f247d94-5720-4a5e-af09-61639affce37-bundle" (OuterVolumeSpecName: "bundle") pod "6f247d94-5720-4a5e-af09-61639affce37" (UID: "6f247d94-5720-4a5e-af09-61639affce37"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:36:47.382421 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:47.382393 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f247d94-5720-4a5e-af09-61639affce37-kube-api-access-kmk2g" (OuterVolumeSpecName: "kube-api-access-kmk2g") pod "6f247d94-5720-4a5e-af09-61639affce37" (UID: "6f247d94-5720-4a5e-af09-61639affce37"). InnerVolumeSpecName "kube-api-access-kmk2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 11:36:47.382990 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:47.382965 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540-kube-api-access-prxtv" (OuterVolumeSpecName: "kube-api-access-prxtv") pod "eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540" (UID: "eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540"). InnerVolumeSpecName "kube-api-access-prxtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 11:36:47.386410 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:47.386385 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540-util" (OuterVolumeSpecName: "util") pod "eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540" (UID: "eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:36:47.386805 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:47.386745 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f247d94-5720-4a5e-af09-61639affce37-util" (OuterVolumeSpecName: "util") pod "6f247d94-5720-4a5e-af09-61639affce37" (UID: "6f247d94-5720-4a5e-af09-61639affce37"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:36:47.480978 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:47.480946 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prxtv\" (UniqueName: \"kubernetes.io/projected/eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540-kube-api-access-prxtv\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:36:47.480978 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:47.480972 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f247d94-5720-4a5e-af09-61639affce37-bundle\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:36:47.480978 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:47.480981 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540-bundle\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:36:47.480978 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:47.480989 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f247d94-5720-4a5e-af09-61639affce37-util\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:36:47.481222 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:47.480996 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540-util\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:36:47.481222 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:47.481003 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmk2g\" (UniqueName: \"kubernetes.io/projected/6f247d94-5720-4a5e-af09-61639affce37-kube-api-access-kmk2g\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:36:48.098918 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:48.098894 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf" Jul 06 11:36:48.099258 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:48.098923 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf" event={"ID":"6f247d94-5720-4a5e-af09-61639affce37","Type":"ContainerDied","Data":"b14c9d29438555a0ea814fd04b409d600ab9e354f3f61f2e551f762709249e69"} Jul 06 11:36:48.099258 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:48.098970 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b14c9d29438555a0ea814fd04b409d600ab9e354f3f61f2e551f762709249e69" Jul 06 11:36:48.101036 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:48.100997 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql" Jul 06 11:36:48.101036 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:48.101011 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql" event={"ID":"eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540","Type":"ContainerDied","Data":"25a6e4c3ce1d6dab7977903210da745d0b338e36982928ab73d2f868be396e9f"} Jul 06 11:36:48.101178 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:48.101057 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25a6e4c3ce1d6dab7977903210da745d0b338e36982928ab73d2f868be396e9f" Jul 06 11:36:48.220739 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:48.220710 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx" Jul 06 11:36:48.388577 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:48.388554 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/969a11c9-a0fa-45d1-b093-4bb6c5fc84b4-bundle\") pod \"969a11c9-a0fa-45d1-b093-4bb6c5fc84b4\" (UID: \"969a11c9-a0fa-45d1-b093-4bb6c5fc84b4\") " Jul 06 11:36:48.388761 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:48.388584 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/969a11c9-a0fa-45d1-b093-4bb6c5fc84b4-util\") pod \"969a11c9-a0fa-45d1-b093-4bb6c5fc84b4\" (UID: \"969a11c9-a0fa-45d1-b093-4bb6c5fc84b4\") " Jul 06 11:36:48.388761 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:48.388639 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtblr\" (UniqueName: \"kubernetes.io/projected/969a11c9-a0fa-45d1-b093-4bb6c5fc84b4-kube-api-access-qtblr\") pod \"969a11c9-a0fa-45d1-b093-4bb6c5fc84b4\" (UID: \"969a11c9-a0fa-45d1-b093-4bb6c5fc84b4\") " Jul 06 11:36:48.389102 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:48.389064 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/969a11c9-a0fa-45d1-b093-4bb6c5fc84b4-bundle" (OuterVolumeSpecName: "bundle") pod "969a11c9-a0fa-45d1-b093-4bb6c5fc84b4" (UID: "969a11c9-a0fa-45d1-b093-4bb6c5fc84b4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:36:48.390804 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:48.390774 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/969a11c9-a0fa-45d1-b093-4bb6c5fc84b4-kube-api-access-qtblr" (OuterVolumeSpecName: "kube-api-access-qtblr") pod "969a11c9-a0fa-45d1-b093-4bb6c5fc84b4" (UID: "969a11c9-a0fa-45d1-b093-4bb6c5fc84b4"). InnerVolumeSpecName "kube-api-access-qtblr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 11:36:48.393940 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:48.393915 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/969a11c9-a0fa-45d1-b093-4bb6c5fc84b4-util" (OuterVolumeSpecName: "util") pod "969a11c9-a0fa-45d1-b093-4bb6c5fc84b4" (UID: "969a11c9-a0fa-45d1-b093-4bb6c5fc84b4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:36:48.489749 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:48.489721 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/969a11c9-a0fa-45d1-b093-4bb6c5fc84b4-bundle\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:36:48.489749 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:48.489744 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/969a11c9-a0fa-45d1-b093-4bb6c5fc84b4-util\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:36:48.489749 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:48.489754 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtblr\" (UniqueName: \"kubernetes.io/projected/969a11c9-a0fa-45d1-b093-4bb6c5fc84b4-kube-api-access-qtblr\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:36:49.112469 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:49.112436 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx" event={"ID":"969a11c9-a0fa-45d1-b093-4bb6c5fc84b4","Type":"ContainerDied","Data":"1ea0e22a9dd79a957daaf43eef1fb566006ab21eaea420efa224e7db5b9c9087"} Jul 06 11:36:49.112469 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:49.112457 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx" Jul 06 11:36:49.112905 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:36:49.112504 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ea0e22a9dd79a957daaf43eef1fb566006ab21eaea420efa224e7db5b9c9087" Jul 06 11:37:09.486910 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.486870 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-ptbjr"] Jul 06 11:37:09.487372 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.487338 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f247d94-5720-4a5e-af09-61639affce37" containerName="util" Jul 06 11:37:09.487372 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.487356 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f247d94-5720-4a5e-af09-61639affce37" containerName="util" Jul 06 11:37:09.487502 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.487375 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="969a11c9-a0fa-45d1-b093-4bb6c5fc84b4" containerName="pull" Jul 06 11:37:09.487502 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.487384 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="969a11c9-a0fa-45d1-b093-4bb6c5fc84b4" containerName="pull" Jul 06 11:37:09.487502 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.487396 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6085760-c8a0-4f8c-9494-e86fd58675ac" containerName="extract" Jul 06 11:37:09.487502 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.487405 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6085760-c8a0-4f8c-9494-e86fd58675ac" containerName="extract" Jul 06 11:37:09.487502 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.487413 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540" containerName="extract" Jul 06 11:37:09.487502 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.487422 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540" containerName="extract" Jul 06 11:37:09.487502 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.487435 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f247d94-5720-4a5e-af09-61639affce37" containerName="pull" Jul 06 11:37:09.487502 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.487443 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f247d94-5720-4a5e-af09-61639affce37" containerName="pull" Jul 06 11:37:09.487502 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.487463 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540" containerName="util" Jul 06 11:37:09.487502 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.487471 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540" containerName="util" Jul 06 11:37:09.487502 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.487481 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6085760-c8a0-4f8c-9494-e86fd58675ac" containerName="pull" Jul 06 11:37:09.487502 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.487489 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6085760-c8a0-4f8c-9494-e86fd58675ac" containerName="pull" Jul 06 11:37:09.487502 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.487505 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6085760-c8a0-4f8c-9494-e86fd58675ac" containerName="util" Jul 06 11:37:09.488142 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.487513 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6085760-c8a0-4f8c-9494-e86fd58675ac" containerName="util" Jul 06 11:37:09.488142 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.487522 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540" containerName="pull" Jul 06 11:37:09.488142 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.487530 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540" containerName="pull" Jul 06 11:37:09.488142 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.487539 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f247d94-5720-4a5e-af09-61639affce37" containerName="extract" Jul 06 11:37:09.488142 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.487547 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f247d94-5720-4a5e-af09-61639affce37" containerName="extract" Jul 06 11:37:09.488142 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.487557 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="969a11c9-a0fa-45d1-b093-4bb6c5fc84b4" containerName="util" Jul 06 11:37:09.488142 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.487565 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="969a11c9-a0fa-45d1-b093-4bb6c5fc84b4" containerName="util" Jul 06 11:37:09.488142 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.487573 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="969a11c9-a0fa-45d1-b093-4bb6c5fc84b4" containerName="extract" Jul 06 11:37:09.488142 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.487580 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="969a11c9-a0fa-45d1-b093-4bb6c5fc84b4" containerName="extract" Jul 06 11:37:09.488142 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.487669 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="969a11c9-a0fa-45d1-b093-4bb6c5fc84b4" containerName="extract" Jul 06 11:37:09.488142 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.487708 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f247d94-5720-4a5e-af09-61639affce37" containerName="extract" Jul 06 11:37:09.488142 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.487718 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540" containerName="extract" Jul 06 11:37:09.488142 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.487727 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6085760-c8a0-4f8c-9494-e86fd58675ac" containerName="extract" Jul 06 11:37:09.491092 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.491072 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ptbjr" Jul 06 11:37:09.494269 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.494241 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 06 11:37:09.494269 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.494241 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 11:37:09.494433 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.494269 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-kxwt5\"" Jul 06 11:37:09.495446 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.495431 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 11:37:09.500059 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.500030 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-ptbjr"] Jul 06 11:37:09.552945 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.552912 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thm52\" (UniqueName: \"kubernetes.io/projected/8565fb29-137e-4199-b150-dc34077ab6b8-kube-api-access-thm52\") pod \"dns-operator-controller-manager-864dd78c4c-ptbjr\" (UID: \"8565fb29-137e-4199-b150-dc34077ab6b8\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ptbjr" Jul 06 11:37:09.653954 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.653921 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thm52\" (UniqueName: \"kubernetes.io/projected/8565fb29-137e-4199-b150-dc34077ab6b8-kube-api-access-thm52\") pod \"dns-operator-controller-manager-864dd78c4c-ptbjr\" (UID: \"8565fb29-137e-4199-b150-dc34077ab6b8\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ptbjr" Jul 06 11:37:09.664239 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.664212 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thm52\" (UniqueName: \"kubernetes.io/projected/8565fb29-137e-4199-b150-dc34077ab6b8-kube-api-access-thm52\") pod \"dns-operator-controller-manager-864dd78c4c-ptbjr\" (UID: \"8565fb29-137e-4199-b150-dc34077ab6b8\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ptbjr" Jul 06 11:37:09.802407 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.802315 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ptbjr" Jul 06 11:37:09.931216 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:09.931188 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-ptbjr"] Jul 06 11:37:09.933611 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:37:09.933583 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8565fb29_137e_4199_b150_dc34077ab6b8.slice/crio-c1aed441ebc6093eba8baa497b0d2b626d9fab4b6c687591ee0da0a5b72ff9d9 WatchSource:0}: Error finding container c1aed441ebc6093eba8baa497b0d2b626d9fab4b6c687591ee0da0a5b72ff9d9: Status 404 returned error can't find the container with id c1aed441ebc6093eba8baa497b0d2b626d9fab4b6c687591ee0da0a5b72ff9d9 Jul 06 11:37:10.215136 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:10.215098 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ptbjr" event={"ID":"8565fb29-137e-4199-b150-dc34077ab6b8","Type":"ContainerStarted","Data":"c1aed441ebc6093eba8baa497b0d2b626d9fab4b6c687591ee0da0a5b72ff9d9"} Jul 06 11:37:12.227620 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:12.227585 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ptbjr" event={"ID":"8565fb29-137e-4199-b150-dc34077ab6b8","Type":"ContainerStarted","Data":"842a73214b513369964c813fcf41ef61e74b4112986bfd068e242f0ac2391518"} Jul 06 11:37:12.228046 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:12.227708 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ptbjr" Jul 06 11:37:12.250416 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:12.250367 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ptbjr" podStartSLOduration=1.484229445 podStartE2EDuration="3.250352912s" totalImagesPullingTime="1.766123467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:37:12.24891494 +0000 UTC m=+530.728713955" watchObservedRunningTime="2026-07-06 11:37:12.250352912 +0000 UTC m=+530.730151929" Jul 06 11:37:18.826792 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:18.826713 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9k4gr"] Jul 06 11:37:18.831549 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:18.831528 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9k4gr" Jul 06 11:37:18.834533 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:18.834515 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qlxfr\"" Jul 06 11:37:18.843349 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:18.843326 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9k4gr"] Jul 06 11:37:18.934584 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:18.934552 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9tpg\" (UniqueName: \"kubernetes.io/projected/61b231f5-3f7a-4194-bb88-aedf63314d5f-kube-api-access-k9tpg\") pod \"kuadrant-operator-controller-manager-86878c769f-9k4gr\" (UID: \"61b231f5-3f7a-4194-bb88-aedf63314d5f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9k4gr" Jul 06 11:37:18.934795 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:18.934592 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61b231f5-3f7a-4194-bb88-aedf63314d5f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9k4gr\" (UID: \"61b231f5-3f7a-4194-bb88-aedf63314d5f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9k4gr" Jul 06 11:37:19.035807 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:19.035772 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61b231f5-3f7a-4194-bb88-aedf63314d5f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9k4gr\" (UID: \"61b231f5-3f7a-4194-bb88-aedf63314d5f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9k4gr" Jul 06 11:37:19.035978 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:19.035872 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9tpg\" (UniqueName: \"kubernetes.io/projected/61b231f5-3f7a-4194-bb88-aedf63314d5f-kube-api-access-k9tpg\") pod \"kuadrant-operator-controller-manager-86878c769f-9k4gr\" (UID: \"61b231f5-3f7a-4194-bb88-aedf63314d5f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9k4gr" Jul 06 11:37:19.036160 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:19.036142 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61b231f5-3f7a-4194-bb88-aedf63314d5f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9k4gr\" (UID: \"61b231f5-3f7a-4194-bb88-aedf63314d5f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9k4gr" Jul 06 11:37:19.056362 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:19.056329 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9tpg\" (UniqueName: \"kubernetes.io/projected/61b231f5-3f7a-4194-bb88-aedf63314d5f-kube-api-access-k9tpg\") pod \"kuadrant-operator-controller-manager-86878c769f-9k4gr\" (UID: \"61b231f5-3f7a-4194-bb88-aedf63314d5f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9k4gr" Jul 06 11:37:19.142718 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:19.142659 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9k4gr" Jul 06 11:37:19.279043 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:19.279019 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9k4gr"] Jul 06 11:37:19.280721 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:37:19.280671 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b231f5_3f7a_4194_bb88_aedf63314d5f.slice/crio-bb65e139786c178b0055f58d220d8db6f31d3171127adc16a932acb3ecda6fce WatchSource:0}: Error finding container bb65e139786c178b0055f58d220d8db6f31d3171127adc16a932acb3ecda6fce: Status 404 returned error can't find the container with id bb65e139786c178b0055f58d220d8db6f31d3171127adc16a932acb3ecda6fce Jul 06 11:37:20.271297 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:20.271259 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9k4gr" event={"ID":"61b231f5-3f7a-4194-bb88-aedf63314d5f","Type":"ContainerStarted","Data":"bb65e139786c178b0055f58d220d8db6f31d3171127adc16a932acb3ecda6fce"} Jul 06 11:37:23.235332 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:23.235302 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-ptbjr" Jul 06 11:37:24.294635 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:24.294598 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9k4gr" event={"ID":"61b231f5-3f7a-4194-bb88-aedf63314d5f","Type":"ContainerStarted","Data":"ac83a0d689724ef53cc793e42af5d02c1ab5ec3d233aed77503aaac393829b05"} Jul 06 11:37:24.295032 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:24.294743 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9k4gr" Jul 06 11:37:24.318432 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:24.318374 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9k4gr" podStartSLOduration=1.9196490019999999 podStartE2EDuration="6.318360625s" totalImagesPullingTime="4.398711623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:37:24.316323208 +0000 UTC m=+542.796122224" watchObservedRunningTime="2026-07-06 11:37:24.318360625 +0000 UTC m=+542.798159688" Jul 06 11:37:35.302507 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:37:35.302471 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9k4gr" Jul 06 11:38:36.176573 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:36.176541 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5cd666f5bf-b6nhg"] Jul 06 11:38:36.179989 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:36.179966 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd666f5bf-b6nhg" Jul 06 11:38:36.182899 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:36.182869 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 11:38:36.182899 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:36.182888 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 11:38:36.183073 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:36.183035 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vxnbk\"" Jul 06 11:38:36.191384 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:36.191361 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cd666f5bf-b6nhg"] Jul 06 11:38:36.317442 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:36.317404 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9880124-f1fc-4a85-b3e4-f0539e8ea109-webhook-cert\") pod \"maas-controller-5cd666f5bf-b6nhg\" (UID: \"d9880124-f1fc-4a85-b3e4-f0539e8ea109\") " pod="opendatahub/maas-controller-5cd666f5bf-b6nhg" Jul 06 11:38:36.317632 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:36.317485 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcsmx\" (UniqueName: \"kubernetes.io/projected/d9880124-f1fc-4a85-b3e4-f0539e8ea109-kube-api-access-jcsmx\") pod \"maas-controller-5cd666f5bf-b6nhg\" (UID: \"d9880124-f1fc-4a85-b3e4-f0539e8ea109\") " pod="opendatahub/maas-controller-5cd666f5bf-b6nhg" Jul 06 11:38:36.358798 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:36.358768 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5fc875b447-7qn62"] Jul 06 11:38:36.362148 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:36.362126 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fc875b447-7qn62" Jul 06 11:38:36.373023 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:36.372995 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fc875b447-7qn62"] Jul 06 11:38:36.418403 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:36.418360 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9880124-f1fc-4a85-b3e4-f0539e8ea109-webhook-cert\") pod \"maas-controller-5cd666f5bf-b6nhg\" (UID: \"d9880124-f1fc-4a85-b3e4-f0539e8ea109\") " pod="opendatahub/maas-controller-5cd666f5bf-b6nhg" Jul 06 11:38:36.418555 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:36.418454 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcsmx\" (UniqueName: \"kubernetes.io/projected/d9880124-f1fc-4a85-b3e4-f0539e8ea109-kube-api-access-jcsmx\") pod \"maas-controller-5cd666f5bf-b6nhg\" (UID: \"d9880124-f1fc-4a85-b3e4-f0539e8ea109\") " pod="opendatahub/maas-controller-5cd666f5bf-b6nhg" Jul 06 11:38:36.418555 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:38:36.418522 2568 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 06 11:38:36.418651 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:38:36.418592 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9880124-f1fc-4a85-b3e4-f0539e8ea109-webhook-cert podName:d9880124-f1fc-4a85-b3e4-f0539e8ea109 nodeName:}" failed. No retries permitted until 2026-07-06 11:38:36.918572524 +0000 UTC m=+615.398371532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/d9880124-f1fc-4a85-b3e4-f0539e8ea109-webhook-cert") pod "maas-controller-5cd666f5bf-b6nhg" (UID: "d9880124-f1fc-4a85-b3e4-f0539e8ea109") : secret "maas-controller-webhook-cert" not found Jul 06 11:38:36.436359 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:36.436287 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcsmx\" (UniqueName: \"kubernetes.io/projected/d9880124-f1fc-4a85-b3e4-f0539e8ea109-kube-api-access-jcsmx\") pod \"maas-controller-5cd666f5bf-b6nhg\" (UID: \"d9880124-f1fc-4a85-b3e4-f0539e8ea109\") " pod="opendatahub/maas-controller-5cd666f5bf-b6nhg" Jul 06 11:38:36.519122 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:36.519077 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b66961a3-8287-4d22-8198-c44fd2769504-webhook-cert\") pod \"maas-controller-5fc875b447-7qn62\" (UID: \"b66961a3-8287-4d22-8198-c44fd2769504\") " pod="opendatahub/maas-controller-5fc875b447-7qn62" Jul 06 11:38:36.519304 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:36.519230 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gptjf\" (UniqueName: \"kubernetes.io/projected/b66961a3-8287-4d22-8198-c44fd2769504-kube-api-access-gptjf\") pod \"maas-controller-5fc875b447-7qn62\" (UID: \"b66961a3-8287-4d22-8198-c44fd2769504\") " pod="opendatahub/maas-controller-5fc875b447-7qn62" Jul 06 11:38:36.620646 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:36.620604 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gptjf\" (UniqueName: \"kubernetes.io/projected/b66961a3-8287-4d22-8198-c44fd2769504-kube-api-access-gptjf\") pod \"maas-controller-5fc875b447-7qn62\" (UID: \"b66961a3-8287-4d22-8198-c44fd2769504\") " pod="opendatahub/maas-controller-5fc875b447-7qn62" Jul 06 11:38:36.620646 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:36.620646 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b66961a3-8287-4d22-8198-c44fd2769504-webhook-cert\") pod \"maas-controller-5fc875b447-7qn62\" (UID: \"b66961a3-8287-4d22-8198-c44fd2769504\") " pod="opendatahub/maas-controller-5fc875b447-7qn62" Jul 06 11:38:36.620905 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:38:36.620809 2568 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 06 11:38:36.620905 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:38:36.620892 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b66961a3-8287-4d22-8198-c44fd2769504-webhook-cert podName:b66961a3-8287-4d22-8198-c44fd2769504 nodeName:}" failed. No retries permitted until 2026-07-06 11:38:37.120872635 +0000 UTC m=+615.600671635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/b66961a3-8287-4d22-8198-c44fd2769504-webhook-cert") pod "maas-controller-5fc875b447-7qn62" (UID: "b66961a3-8287-4d22-8198-c44fd2769504") : secret "maas-controller-webhook-cert" not found Jul 06 11:38:36.632896 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:36.632867 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gptjf\" (UniqueName: \"kubernetes.io/projected/b66961a3-8287-4d22-8198-c44fd2769504-kube-api-access-gptjf\") pod \"maas-controller-5fc875b447-7qn62\" (UID: \"b66961a3-8287-4d22-8198-c44fd2769504\") " pod="opendatahub/maas-controller-5fc875b447-7qn62" Jul 06 11:38:36.922735 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:36.922701 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9880124-f1fc-4a85-b3e4-f0539e8ea109-webhook-cert\") pod \"maas-controller-5cd666f5bf-b6nhg\" (UID: \"d9880124-f1fc-4a85-b3e4-f0539e8ea109\") " pod="opendatahub/maas-controller-5cd666f5bf-b6nhg" Jul 06 11:38:36.922995 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:38:36.922855 2568 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 06 11:38:36.922995 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:38:36.922921 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9880124-f1fc-4a85-b3e4-f0539e8ea109-webhook-cert podName:d9880124-f1fc-4a85-b3e4-f0539e8ea109 nodeName:}" failed. No retries permitted until 2026-07-06 11:38:37.922906122 +0000 UTC m=+616.402705115 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/d9880124-f1fc-4a85-b3e4-f0539e8ea109-webhook-cert") pod "maas-controller-5cd666f5bf-b6nhg" (UID: "d9880124-f1fc-4a85-b3e4-f0539e8ea109") : secret "maas-controller-webhook-cert" not found Jul 06 11:38:37.125227 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:37.125190 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b66961a3-8287-4d22-8198-c44fd2769504-webhook-cert\") pod \"maas-controller-5fc875b447-7qn62\" (UID: \"b66961a3-8287-4d22-8198-c44fd2769504\") " pod="opendatahub/maas-controller-5fc875b447-7qn62" Jul 06 11:38:37.127742 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:37.127718 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b66961a3-8287-4d22-8198-c44fd2769504-webhook-cert\") pod \"maas-controller-5fc875b447-7qn62\" (UID: \"b66961a3-8287-4d22-8198-c44fd2769504\") " pod="opendatahub/maas-controller-5fc875b447-7qn62" Jul 06 11:38:37.272803 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:37.272713 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fc875b447-7qn62" Jul 06 11:38:37.409663 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:37.409638 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fc875b447-7qn62"] Jul 06 11:38:37.411903 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:38:37.411876 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb66961a3_8287_4d22_8198_c44fd2769504.slice/crio-afd20ee210c928d8ada156f234e6ec20406815e49137d7d4a3ab7bf3a978b142 WatchSource:0}: Error finding container afd20ee210c928d8ada156f234e6ec20406815e49137d7d4a3ab7bf3a978b142: Status 404 returned error can't find the container with id afd20ee210c928d8ada156f234e6ec20406815e49137d7d4a3ab7bf3a978b142 Jul 06 11:38:37.669160 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:37.669122 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fc875b447-7qn62" event={"ID":"b66961a3-8287-4d22-8198-c44fd2769504","Type":"ContainerStarted","Data":"afd20ee210c928d8ada156f234e6ec20406815e49137d7d4a3ab7bf3a978b142"} Jul 06 11:38:37.933151 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:37.933050 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9880124-f1fc-4a85-b3e4-f0539e8ea109-webhook-cert\") pod \"maas-controller-5cd666f5bf-b6nhg\" (UID: \"d9880124-f1fc-4a85-b3e4-f0539e8ea109\") " pod="opendatahub/maas-controller-5cd666f5bf-b6nhg" Jul 06 11:38:37.935483 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:37.935458 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9880124-f1fc-4a85-b3e4-f0539e8ea109-webhook-cert\") pod \"maas-controller-5cd666f5bf-b6nhg\" (UID: \"d9880124-f1fc-4a85-b3e4-f0539e8ea109\") " pod="opendatahub/maas-controller-5cd666f5bf-b6nhg" Jul 06 11:38:37.991292 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:37.991254 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd666f5bf-b6nhg" Jul 06 11:38:38.131991 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:38.131963 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cd666f5bf-b6nhg"] Jul 06 11:38:38.133895 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:38:38.133865 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9880124_f1fc_4a85_b3e4_f0539e8ea109.slice/crio-865b4e068565565dba4900e2ac6be8a6d6aaba34962eca994534884a931cd883 WatchSource:0}: Error finding container 865b4e068565565dba4900e2ac6be8a6d6aaba34962eca994534884a931cd883: Status 404 returned error can't find the container with id 865b4e068565565dba4900e2ac6be8a6d6aaba34962eca994534884a931cd883 Jul 06 11:38:38.678178 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:38.678140 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd666f5bf-b6nhg" event={"ID":"d9880124-f1fc-4a85-b3e4-f0539e8ea109","Type":"ContainerStarted","Data":"865b4e068565565dba4900e2ac6be8a6d6aaba34962eca994534884a931cd883"} Jul 06 11:38:40.689750 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:40.689707 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd666f5bf-b6nhg" event={"ID":"d9880124-f1fc-4a85-b3e4-f0539e8ea109","Type":"ContainerStarted","Data":"84408f6af6fbad44789726f3ba181c6d64483bd424900d09528746ce42175832"} Jul 06 11:38:40.690191 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:40.689819 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5cd666f5bf-b6nhg" Jul 06 11:38:40.691105 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:40.691084 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fc875b447-7qn62" event={"ID":"b66961a3-8287-4d22-8198-c44fd2769504","Type":"ContainerStarted","Data":"dee5f66f8412ad0da0846fc57f4beebeae70f7899bb07d8aadcd42c16b78134b"} Jul 06 11:38:40.691220 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:40.691203 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5fc875b447-7qn62" Jul 06 11:38:40.730634 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:40.730577 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5cd666f5bf-b6nhg" podStartSLOduration=3.1557618339999998 podStartE2EDuration="4.730561818s" totalImagesPullingTime="1.574799984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:38:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:38:40.727855157 +0000 UTC m=+619.207654173" watchObservedRunningTime="2026-07-06 11:38:40.730561818 +0000 UTC m=+619.210360833" Jul 06 11:38:40.782788 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:40.782735 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5fc875b447-7qn62" podStartSLOduration=2.492757826 podStartE2EDuration="4.782716736s" totalImagesPullingTime="2.28995891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:38:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:38:40.780618971 +0000 UTC m=+619.260417988" watchObservedRunningTime="2026-07-06 11:38:40.782716736 +0000 UTC m=+619.262515753" Jul 06 11:38:51.702038 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:51.701955 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5cd666f5bf-b6nhg" Jul 06 11:38:51.702418 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:51.702040 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5fc875b447-7qn62" Jul 06 11:38:51.779933 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:51.779895 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cd666f5bf-b6nhg"] Jul 06 11:38:51.780199 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:51.780176 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-5cd666f5bf-b6nhg" podUID="d9880124-f1fc-4a85-b3e4-f0539e8ea109" containerName="manager" containerID="cri-o://84408f6af6fbad44789726f3ba181c6d64483bd424900d09528746ce42175832" gracePeriod=30 Jul 06 11:38:52.029368 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:52.029342 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd666f5bf-b6nhg" Jul 06 11:38:52.059314 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:52.059286 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9880124-f1fc-4a85-b3e4-f0539e8ea109-webhook-cert\") pod \"d9880124-f1fc-4a85-b3e4-f0539e8ea109\" (UID: \"d9880124-f1fc-4a85-b3e4-f0539e8ea109\") " Jul 06 11:38:52.059485 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:52.059328 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcsmx\" (UniqueName: \"kubernetes.io/projected/d9880124-f1fc-4a85-b3e4-f0539e8ea109-kube-api-access-jcsmx\") pod \"d9880124-f1fc-4a85-b3e4-f0539e8ea109\" (UID: \"d9880124-f1fc-4a85-b3e4-f0539e8ea109\") " Jul 06 11:38:52.062186 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:52.062158 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9880124-f1fc-4a85-b3e4-f0539e8ea109-kube-api-access-jcsmx" (OuterVolumeSpecName: "kube-api-access-jcsmx") pod "d9880124-f1fc-4a85-b3e4-f0539e8ea109" (UID: "d9880124-f1fc-4a85-b3e4-f0539e8ea109"). InnerVolumeSpecName "kube-api-access-jcsmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 11:38:52.062316 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:52.062184 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9880124-f1fc-4a85-b3e4-f0539e8ea109-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d9880124-f1fc-4a85-b3e4-f0539e8ea109" (UID: "d9880124-f1fc-4a85-b3e4-f0539e8ea109"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 11:38:52.159972 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:52.159936 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcsmx\" (UniqueName: \"kubernetes.io/projected/d9880124-f1fc-4a85-b3e4-f0539e8ea109-kube-api-access-jcsmx\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:38:52.159972 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:52.159965 2568 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9880124-f1fc-4a85-b3e4-f0539e8ea109-webhook-cert\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:38:52.754471 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:52.754430 2568 generic.go:356] "Generic (PLEG): container finished" podID="d9880124-f1fc-4a85-b3e4-f0539e8ea109" containerID="84408f6af6fbad44789726f3ba181c6d64483bd424900d09528746ce42175832" exitCode=0 Jul 06 11:38:52.754902 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:52.754478 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd666f5bf-b6nhg" Jul 06 11:38:52.754902 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:52.754493 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd666f5bf-b6nhg" event={"ID":"d9880124-f1fc-4a85-b3e4-f0539e8ea109","Type":"ContainerDied","Data":"84408f6af6fbad44789726f3ba181c6d64483bd424900d09528746ce42175832"} Jul 06 11:38:52.754902 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:52.754548 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd666f5bf-b6nhg" event={"ID":"d9880124-f1fc-4a85-b3e4-f0539e8ea109","Type":"ContainerDied","Data":"865b4e068565565dba4900e2ac6be8a6d6aaba34962eca994534884a931cd883"} Jul 06 11:38:52.754902 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:52.754578 2568 scope.go:117] "RemoveContainer" containerID="84408f6af6fbad44789726f3ba181c6d64483bd424900d09528746ce42175832" Jul 06 11:38:52.765767 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:52.765748 2568 scope.go:117] "RemoveContainer" containerID="84408f6af6fbad44789726f3ba181c6d64483bd424900d09528746ce42175832" Jul 06 11:38:52.766048 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:38:52.766030 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84408f6af6fbad44789726f3ba181c6d64483bd424900d09528746ce42175832\": container with ID starting with 84408f6af6fbad44789726f3ba181c6d64483bd424900d09528746ce42175832 not found: ID does not exist" containerID="84408f6af6fbad44789726f3ba181c6d64483bd424900d09528746ce42175832" Jul 06 11:38:52.766094 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:52.766057 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84408f6af6fbad44789726f3ba181c6d64483bd424900d09528746ce42175832"} err="failed to get container status \"84408f6af6fbad44789726f3ba181c6d64483bd424900d09528746ce42175832\": rpc error: code = NotFound desc = could not find container \"84408f6af6fbad44789726f3ba181c6d64483bd424900d09528746ce42175832\": container with ID starting with 84408f6af6fbad44789726f3ba181c6d64483bd424900d09528746ce42175832 not found: ID does not exist" Jul 06 11:38:52.781939 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:52.781908 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cd666f5bf-b6nhg"] Jul 06 11:38:52.787430 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:52.787404 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5cd666f5bf-b6nhg"] Jul 06 11:38:54.052268 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:38:54.052236 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9880124-f1fc-4a85-b3e4-f0539e8ea109" path="/var/lib/kubelet/pods/d9880124-f1fc-4a85-b3e4-f0539e8ea109/volumes" Jul 06 11:39:12.095365 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:12.095334 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-ccdfff8b-g5tjf"] Jul 06 11:39:12.095875 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:12.095699 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9880124-f1fc-4a85-b3e4-f0539e8ea109" containerName="manager" Jul 06 11:39:12.095875 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:12.095715 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9880124-f1fc-4a85-b3e4-f0539e8ea109" containerName="manager" Jul 06 11:39:12.095875 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:12.095782 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9880124-f1fc-4a85-b3e4-f0539e8ea109" containerName="manager" Jul 06 11:39:12.098867 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:12.098847 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-ccdfff8b-g5tjf" Jul 06 11:39:12.103100 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:12.103074 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 06 11:39:12.103225 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:12.103074 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-n9mxk\"" Jul 06 11:39:12.114893 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:12.114868 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-ccdfff8b-g5tjf"] Jul 06 11:39:12.231730 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:12.231668 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16d80f49-fc68-452e-ab42-eb4dd72a1dfe-maas-api-tls\") pod \"maas-api-ccdfff8b-g5tjf\" (UID: \"16d80f49-fc68-452e-ab42-eb4dd72a1dfe\") " pod="opendatahub/maas-api-ccdfff8b-g5tjf" Jul 06 11:39:12.231730 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:12.231730 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d80f49-fc68-452e-ab42-eb4dd72a1dfe-openshift-service-ca\") pod \"maas-api-ccdfff8b-g5tjf\" (UID: \"16d80f49-fc68-452e-ab42-eb4dd72a1dfe\") " pod="opendatahub/maas-api-ccdfff8b-g5tjf" Jul 06 11:39:12.231949 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:12.231793 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b5fs\" (UniqueName: \"kubernetes.io/projected/16d80f49-fc68-452e-ab42-eb4dd72a1dfe-kube-api-access-5b5fs\") pod \"maas-api-ccdfff8b-g5tjf\" (UID: \"16d80f49-fc68-452e-ab42-eb4dd72a1dfe\") " pod="opendatahub/maas-api-ccdfff8b-g5tjf" Jul 06 11:39:12.333147 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:12.333106 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b5fs\" (UniqueName: \"kubernetes.io/projected/16d80f49-fc68-452e-ab42-eb4dd72a1dfe-kube-api-access-5b5fs\") pod \"maas-api-ccdfff8b-g5tjf\" (UID: \"16d80f49-fc68-452e-ab42-eb4dd72a1dfe\") " pod="opendatahub/maas-api-ccdfff8b-g5tjf" Jul 06 11:39:12.333389 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:12.333200 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16d80f49-fc68-452e-ab42-eb4dd72a1dfe-maas-api-tls\") pod \"maas-api-ccdfff8b-g5tjf\" (UID: \"16d80f49-fc68-452e-ab42-eb4dd72a1dfe\") " pod="opendatahub/maas-api-ccdfff8b-g5tjf" Jul 06 11:39:12.333389 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:12.333219 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d80f49-fc68-452e-ab42-eb4dd72a1dfe-openshift-service-ca\") pod \"maas-api-ccdfff8b-g5tjf\" (UID: \"16d80f49-fc68-452e-ab42-eb4dd72a1dfe\") " pod="opendatahub/maas-api-ccdfff8b-g5tjf" Jul 06 11:39:12.333998 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:12.333976 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d80f49-fc68-452e-ab42-eb4dd72a1dfe-openshift-service-ca\") pod \"maas-api-ccdfff8b-g5tjf\" (UID: \"16d80f49-fc68-452e-ab42-eb4dd72a1dfe\") " pod="opendatahub/maas-api-ccdfff8b-g5tjf" Jul 06 11:39:12.335776 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:12.335752 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16d80f49-fc68-452e-ab42-eb4dd72a1dfe-maas-api-tls\") pod \"maas-api-ccdfff8b-g5tjf\" (UID: \"16d80f49-fc68-452e-ab42-eb4dd72a1dfe\") " pod="opendatahub/maas-api-ccdfff8b-g5tjf" Jul 06 11:39:12.346172 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:12.346119 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b5fs\" (UniqueName: \"kubernetes.io/projected/16d80f49-fc68-452e-ab42-eb4dd72a1dfe-kube-api-access-5b5fs\") pod \"maas-api-ccdfff8b-g5tjf\" (UID: \"16d80f49-fc68-452e-ab42-eb4dd72a1dfe\") " pod="opendatahub/maas-api-ccdfff8b-g5tjf" Jul 06 11:39:12.409520 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:12.409486 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-ccdfff8b-g5tjf" Jul 06 11:39:12.566839 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:12.566807 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-ccdfff8b-g5tjf"] Jul 06 11:39:12.571201 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:39:12.571170 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d80f49_fc68_452e_ab42_eb4dd72a1dfe.slice/crio-33b9d74165319446c4481a20cac9013e334dc0b06e614b7463033a961f5ac3c9 WatchSource:0}: Error finding container 33b9d74165319446c4481a20cac9013e334dc0b06e614b7463033a961f5ac3c9: Status 404 returned error can't find the container with id 33b9d74165319446c4481a20cac9013e334dc0b06e614b7463033a961f5ac3c9 Jul 06 11:39:12.572516 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:12.572499 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 11:39:12.860179 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:12.860145 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-ccdfff8b-g5tjf" event={"ID":"16d80f49-fc68-452e-ab42-eb4dd72a1dfe","Type":"ContainerStarted","Data":"33b9d74165319446c4481a20cac9013e334dc0b06e614b7463033a961f5ac3c9"} Jul 06 11:39:14.872467 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:14.872430 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-ccdfff8b-g5tjf" event={"ID":"16d80f49-fc68-452e-ab42-eb4dd72a1dfe","Type":"ContainerStarted","Data":"30df482ae8aba92b54d5c4ccb3171e65df28df265132bb6f991e7c100bc30cc5"} Jul 06 11:39:14.872893 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:14.872548 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-ccdfff8b-g5tjf" Jul 06 11:39:14.892304 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:14.892252 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-ccdfff8b-g5tjf" podStartSLOduration=1.430599601 podStartE2EDuration="2.892235537s" totalImagesPullingTime="1.461635936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:39:14.891149223 +0000 UTC m=+653.370948232" watchObservedRunningTime="2026-07-06 11:39:14.892235537 +0000 UTC m=+653.372034552" Jul 06 11:39:20.883492 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:20.883461 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-ccdfff8b-g5tjf" Jul 06 11:39:29.503407 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:29.503365 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6"] Jul 06 11:39:29.507769 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:29.507747 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6" Jul 06 11:39:29.510966 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:29.510940 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 11:39:29.511131 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:29.511110 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 11:39:29.512230 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:29.512216 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lkq6q\"" Jul 06 11:39:29.512313 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:29.512296 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 06 11:39:29.518652 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:29.518629 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6"] Jul 06 11:39:29.592746 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:29.592713 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/746dfe16-0774-46e0-848d-3652274bba02-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6\" (UID: \"746dfe16-0774-46e0-848d-3652274bba02\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6" Jul 06 11:39:29.592746 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:29.592750 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/746dfe16-0774-46e0-848d-3652274bba02-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6\" (UID: \"746dfe16-0774-46e0-848d-3652274bba02\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6" Jul 06 11:39:29.593015 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:29.592775 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kplw\" (UniqueName: \"kubernetes.io/projected/746dfe16-0774-46e0-848d-3652274bba02-kube-api-access-5kplw\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6\" (UID: \"746dfe16-0774-46e0-848d-3652274bba02\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6" Jul 06 11:39:29.593015 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:29.592843 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/746dfe16-0774-46e0-848d-3652274bba02-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6\" (UID: \"746dfe16-0774-46e0-848d-3652274bba02\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6" Jul 06 11:39:29.593015 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:29.592881 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/746dfe16-0774-46e0-848d-3652274bba02-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6\" (UID: \"746dfe16-0774-46e0-848d-3652274bba02\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6" Jul 06 11:39:29.593015 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:29.592926 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/746dfe16-0774-46e0-848d-3652274bba02-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6\" (UID: \"746dfe16-0774-46e0-848d-3652274bba02\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6" Jul 06 11:39:29.693879 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:29.693844 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/746dfe16-0774-46e0-848d-3652274bba02-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6\" (UID: \"746dfe16-0774-46e0-848d-3652274bba02\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6" Jul 06 11:39:29.693879 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:29.693882 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/746dfe16-0774-46e0-848d-3652274bba02-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6\" (UID: \"746dfe16-0774-46e0-848d-3652274bba02\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6" Jul 06 11:39:29.694128 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:29.693902 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kplw\" (UniqueName: \"kubernetes.io/projected/746dfe16-0774-46e0-848d-3652274bba02-kube-api-access-5kplw\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6\" (UID: \"746dfe16-0774-46e0-848d-3652274bba02\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6" Jul 06 11:39:29.694128 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:29.693923 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/746dfe16-0774-46e0-848d-3652274bba02-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6\" (UID: \"746dfe16-0774-46e0-848d-3652274bba02\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6" Jul 06 11:39:29.694128 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:29.693944 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/746dfe16-0774-46e0-848d-3652274bba02-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6\" (UID: \"746dfe16-0774-46e0-848d-3652274bba02\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6" Jul 06 11:39:29.694128 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:29.693986 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/746dfe16-0774-46e0-848d-3652274bba02-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6\" (UID: \"746dfe16-0774-46e0-848d-3652274bba02\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6" Jul 06 11:39:29.694332 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:29.694280 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/746dfe16-0774-46e0-848d-3652274bba02-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6\" (UID: \"746dfe16-0774-46e0-848d-3652274bba02\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6" Jul 06 11:39:29.694332 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:29.694313 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/746dfe16-0774-46e0-848d-3652274bba02-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6\" (UID: \"746dfe16-0774-46e0-848d-3652274bba02\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6" Jul 06 11:39:29.694499 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:29.694390 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/746dfe16-0774-46e0-848d-3652274bba02-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6\" (UID: \"746dfe16-0774-46e0-848d-3652274bba02\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6" Jul 06 11:39:29.696417 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:29.696385 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/746dfe16-0774-46e0-848d-3652274bba02-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6\" (UID: \"746dfe16-0774-46e0-848d-3652274bba02\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6" Jul 06 11:39:29.696591 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:29.696569 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/746dfe16-0774-46e0-848d-3652274bba02-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6\" (UID: \"746dfe16-0774-46e0-848d-3652274bba02\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6" Jul 06 11:39:29.702294 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:29.702272 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kplw\" (UniqueName: \"kubernetes.io/projected/746dfe16-0774-46e0-848d-3652274bba02-kube-api-access-5kplw\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6\" (UID: \"746dfe16-0774-46e0-848d-3652274bba02\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6" Jul 06 11:39:29.819617 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:29.819519 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6" Jul 06 11:39:29.960807 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:29.960776 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6"] Jul 06 11:39:29.963017 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:39:29.962986 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746dfe16_0774_46e0_848d_3652274bba02.slice/crio-3d051108d8adf6ad95e20c56b3e851e3e60a72880556ef18672056625b956a02 WatchSource:0}: Error finding container 3d051108d8adf6ad95e20c56b3e851e3e60a72880556ef18672056625b956a02: Status 404 returned error can't find the container with id 3d051108d8adf6ad95e20c56b3e851e3e60a72880556ef18672056625b956a02 Jul 06 11:39:30.950732 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:30.950665 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6" event={"ID":"746dfe16-0774-46e0-848d-3652274bba02","Type":"ContainerStarted","Data":"3d051108d8adf6ad95e20c56b3e851e3e60a72880556ef18672056625b956a02"} Jul 06 11:39:31.957725 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:31.957686 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6" event={"ID":"746dfe16-0774-46e0-848d-3652274bba02","Type":"ContainerStarted","Data":"2d775f4ba24b4441916892d758d97c890dee161338d97aecc44afaf1c3a2c546"} Jul 06 11:39:31.981941 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:31.981889 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6" podStartSLOduration=1.815767486 podStartE2EDuration="2.981871591s" totalImagesPullingTime="1.166104105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:39:31.979838565 +0000 UTC m=+670.459637582" watchObservedRunningTime="2026-07-06 11:39:31.981871591 +0000 UTC m=+670.461670606" Jul 06 11:39:39.819848 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:39.819808 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6" Jul 06 11:39:39.820274 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:39.819882 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6" Jul 06 11:39:39.832922 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:39.832879 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6" Jul 06 11:39:40.008959 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:40.008924 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6" Jul 06 11:39:56.404996 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:56.404963 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8"] Jul 06 11:39:56.408331 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:56.408314 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8" Jul 06 11:39:56.412440 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:56.412412 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 06 11:39:56.423961 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:56.423931 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8"] Jul 06 11:39:56.513596 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:56.513555 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8\" (UID: \"3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8" Jul 06 11:39:56.513809 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:56.513615 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8\" (UID: \"3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8" Jul 06 11:39:56.513809 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:56.513701 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8\" (UID: \"3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8" Jul 06 11:39:56.513809 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:56.513745 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8\" (UID: \"3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8" Jul 06 11:39:56.513809 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:56.513772 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tjvn\" (UniqueName: \"kubernetes.io/projected/3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef-kube-api-access-6tjvn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8\" (UID: \"3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8" Jul 06 11:39:56.513809 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:56.513790 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8\" (UID: \"3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8" Jul 06 11:39:56.614720 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:56.614667 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8\" (UID: \"3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8" Jul 06 11:39:56.614901 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:56.614741 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8\" (UID: \"3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8" Jul 06 11:39:56.614901 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:56.614771 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8\" (UID: \"3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8" Jul 06 11:39:56.614901 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:56.614874 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8\" (UID: \"3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8" Jul 06 11:39:56.615071 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:56.614924 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tjvn\" (UniqueName: \"kubernetes.io/projected/3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef-kube-api-access-6tjvn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8\" (UID: \"3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8" Jul 06 11:39:56.615071 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:56.614957 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8\" (UID: \"3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8" Jul 06 11:39:56.615175 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:56.615113 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8\" (UID: \"3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8" Jul 06 11:39:56.615175 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:56.615153 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8\" (UID: \"3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8" Jul 06 11:39:56.615362 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:56.615335 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8\" (UID: \"3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8" Jul 06 11:39:56.617036 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:56.617006 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8\" (UID: \"3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8" Jul 06 11:39:56.617261 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:56.617240 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8\" (UID: \"3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8" Jul 06 11:39:56.625123 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:56.625101 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tjvn\" (UniqueName: \"kubernetes.io/projected/3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef-kube-api-access-6tjvn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8\" (UID: \"3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8" Jul 06 11:39:56.724372 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:56.724280 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8" Jul 06 11:39:56.860428 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:56.860399 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8"] Jul 06 11:39:56.862062 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:39:56.862036 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e3c1fdd_0b43_4bd8_b498_8a3f0df14bef.slice/crio-2fefc11f408f23b59d1a8b302ca5e4e2489f7472481f9a5b76e7d5a0a188361f WatchSource:0}: Error finding container 2fefc11f408f23b59d1a8b302ca5e4e2489f7472481f9a5b76e7d5a0a188361f: Status 404 returned error can't find the container with id 2fefc11f408f23b59d1a8b302ca5e4e2489f7472481f9a5b76e7d5a0a188361f Jul 06 11:39:57.086204 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:57.086124 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8" event={"ID":"3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef","Type":"ContainerStarted","Data":"2fefc11f408f23b59d1a8b302ca5e4e2489f7472481f9a5b76e7d5a0a188361f"} Jul 06 11:39:58.091890 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:58.091855 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8" event={"ID":"3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef","Type":"ContainerStarted","Data":"8fb6ca6324a83fa3fc39b9650c87b7b573136e53d5f6fc9eb2ed0118a78ff1fe"} Jul 06 11:39:58.114594 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:39:58.114536 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8" podStartSLOduration=1.8677433730000002 podStartE2EDuration="2.114521453s" totalImagesPullingTime="246.77808ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:39:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:39:58.11212721 +0000 UTC m=+696.591926231" watchObservedRunningTime="2026-07-06 11:39:58.114521453 +0000 UTC m=+696.594320469" Jul 06 11:40:06.725223 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:40:06.725182 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8" Jul 06 11:40:06.725662 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:40:06.725301 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8" Jul 06 11:40:06.737407 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:40:06.737381 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8" Jul 06 11:40:07.152583 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:40:07.152557 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8" Jul 06 11:43:13.650766 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:13.650732 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fc875b447-7qn62"] Jul 06 11:43:13.651443 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:13.651009 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-5fc875b447-7qn62" podUID="b66961a3-8287-4d22-8198-c44fd2769504" containerName="manager" containerID="cri-o://dee5f66f8412ad0da0846fc57f4beebeae70f7899bb07d8aadcd42c16b78134b" gracePeriod=30 Jul 06 11:43:13.899320 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:13.899294 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fc875b447-7qn62" Jul 06 11:43:14.019963 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:14.019875 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b66961a3-8287-4d22-8198-c44fd2769504-webhook-cert\") pod \"b66961a3-8287-4d22-8198-c44fd2769504\" (UID: \"b66961a3-8287-4d22-8198-c44fd2769504\") " Jul 06 11:43:14.019963 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:14.019922 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gptjf\" (UniqueName: \"kubernetes.io/projected/b66961a3-8287-4d22-8198-c44fd2769504-kube-api-access-gptjf\") pod \"b66961a3-8287-4d22-8198-c44fd2769504\" (UID: \"b66961a3-8287-4d22-8198-c44fd2769504\") " Jul 06 11:43:14.021883 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:14.021855 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b66961a3-8287-4d22-8198-c44fd2769504-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b66961a3-8287-4d22-8198-c44fd2769504" (UID: "b66961a3-8287-4d22-8198-c44fd2769504"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 11:43:14.021981 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:14.021943 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66961a3-8287-4d22-8198-c44fd2769504-kube-api-access-gptjf" (OuterVolumeSpecName: "kube-api-access-gptjf") pod "b66961a3-8287-4d22-8198-c44fd2769504" (UID: "b66961a3-8287-4d22-8198-c44fd2769504"). InnerVolumeSpecName "kube-api-access-gptjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 11:43:14.120762 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:14.120723 2568 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b66961a3-8287-4d22-8198-c44fd2769504-webhook-cert\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:43:14.120762 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:14.120752 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gptjf\" (UniqueName: \"kubernetes.io/projected/b66961a3-8287-4d22-8198-c44fd2769504-kube-api-access-gptjf\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:43:14.126538 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:14.126508 2568 generic.go:356] "Generic (PLEG): container finished" podID="b66961a3-8287-4d22-8198-c44fd2769504" containerID="dee5f66f8412ad0da0846fc57f4beebeae70f7899bb07d8aadcd42c16b78134b" exitCode=0 Jul 06 11:43:14.126721 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:14.126556 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fc875b447-7qn62" Jul 06 11:43:14.126721 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:14.126587 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fc875b447-7qn62" event={"ID":"b66961a3-8287-4d22-8198-c44fd2769504","Type":"ContainerDied","Data":"dee5f66f8412ad0da0846fc57f4beebeae70f7899bb07d8aadcd42c16b78134b"} Jul 06 11:43:14.126721 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:14.126642 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fc875b447-7qn62" event={"ID":"b66961a3-8287-4d22-8198-c44fd2769504","Type":"ContainerDied","Data":"afd20ee210c928d8ada156f234e6ec20406815e49137d7d4a3ab7bf3a978b142"} Jul 06 11:43:14.126721 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:14.126663 2568 scope.go:117] "RemoveContainer" containerID="dee5f66f8412ad0da0846fc57f4beebeae70f7899bb07d8aadcd42c16b78134b" Jul 06 11:43:14.135565 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:14.135551 2568 scope.go:117] "RemoveContainer" containerID="dee5f66f8412ad0da0846fc57f4beebeae70f7899bb07d8aadcd42c16b78134b" Jul 06 11:43:14.135850 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:43:14.135829 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dee5f66f8412ad0da0846fc57f4beebeae70f7899bb07d8aadcd42c16b78134b\": container with ID starting with dee5f66f8412ad0da0846fc57f4beebeae70f7899bb07d8aadcd42c16b78134b not found: ID does not exist" containerID="dee5f66f8412ad0da0846fc57f4beebeae70f7899bb07d8aadcd42c16b78134b" Jul 06 11:43:14.135946 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:14.135857 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dee5f66f8412ad0da0846fc57f4beebeae70f7899bb07d8aadcd42c16b78134b"} err="failed to get container status \"dee5f66f8412ad0da0846fc57f4beebeae70f7899bb07d8aadcd42c16b78134b\": rpc error: code = NotFound desc = could not find container \"dee5f66f8412ad0da0846fc57f4beebeae70f7899bb07d8aadcd42c16b78134b\": container with ID starting with dee5f66f8412ad0da0846fc57f4beebeae70f7899bb07d8aadcd42c16b78134b not found: ID does not exist" Jul 06 11:43:14.151081 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:14.151020 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fc875b447-7qn62"] Jul 06 11:43:14.154851 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:14.154826 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5fc875b447-7qn62"] Jul 06 11:43:14.968375 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:14.968288 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5fc875b447-dlqtv"] Jul 06 11:43:14.968887 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:14.968815 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b66961a3-8287-4d22-8198-c44fd2769504" containerName="manager" Jul 06 11:43:14.968887 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:14.968833 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b66961a3-8287-4d22-8198-c44fd2769504" containerName="manager" Jul 06 11:43:14.968982 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:14.968924 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b66961a3-8287-4d22-8198-c44fd2769504" containerName="manager" Jul 06 11:43:14.973404 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:14.973387 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fc875b447-dlqtv" Jul 06 11:43:14.977774 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:14.977745 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 11:43:14.977774 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:14.977749 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 11:43:14.978040 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:14.977749 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vxnbk\"" Jul 06 11:43:14.994256 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:14.994224 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fc875b447-dlqtv"] Jul 06 11:43:15.028246 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:15.028213 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27876b93-9b29-42c7-b21e-902222ff3daa-webhook-cert\") pod \"maas-controller-5fc875b447-dlqtv\" (UID: \"27876b93-9b29-42c7-b21e-902222ff3daa\") " pod="opendatahub/maas-controller-5fc875b447-dlqtv" Jul 06 11:43:15.028405 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:15.028311 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t6jl\" (UniqueName: \"kubernetes.io/projected/27876b93-9b29-42c7-b21e-902222ff3daa-kube-api-access-6t6jl\") pod \"maas-controller-5fc875b447-dlqtv\" (UID: \"27876b93-9b29-42c7-b21e-902222ff3daa\") " pod="opendatahub/maas-controller-5fc875b447-dlqtv" Jul 06 11:43:15.128897 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:15.128857 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t6jl\" (UniqueName: \"kubernetes.io/projected/27876b93-9b29-42c7-b21e-902222ff3daa-kube-api-access-6t6jl\") pod \"maas-controller-5fc875b447-dlqtv\" (UID: \"27876b93-9b29-42c7-b21e-902222ff3daa\") " pod="opendatahub/maas-controller-5fc875b447-dlqtv" Jul 06 11:43:15.129099 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:15.128931 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27876b93-9b29-42c7-b21e-902222ff3daa-webhook-cert\") pod \"maas-controller-5fc875b447-dlqtv\" (UID: \"27876b93-9b29-42c7-b21e-902222ff3daa\") " pod="opendatahub/maas-controller-5fc875b447-dlqtv" Jul 06 11:43:15.131513 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:15.131483 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27876b93-9b29-42c7-b21e-902222ff3daa-webhook-cert\") pod \"maas-controller-5fc875b447-dlqtv\" (UID: \"27876b93-9b29-42c7-b21e-902222ff3daa\") " pod="opendatahub/maas-controller-5fc875b447-dlqtv" Jul 06 11:43:15.139692 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:15.139656 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t6jl\" (UniqueName: \"kubernetes.io/projected/27876b93-9b29-42c7-b21e-902222ff3daa-kube-api-access-6t6jl\") pod \"maas-controller-5fc875b447-dlqtv\" (UID: \"27876b93-9b29-42c7-b21e-902222ff3daa\") " pod="opendatahub/maas-controller-5fc875b447-dlqtv" Jul 06 11:43:15.283984 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:15.283881 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fc875b447-dlqtv" Jul 06 11:43:15.448772 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:15.448742 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fc875b447-dlqtv"] Jul 06 11:43:15.450005 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:43:15.449979 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27876b93_9b29_42c7_b21e_902222ff3daa.slice/crio-d8db89a27b7e4450000c8a669ca37cc356b1a2ff5398319fab8b33b7786f5b9d WatchSource:0}: Error finding container d8db89a27b7e4450000c8a669ca37cc356b1a2ff5398319fab8b33b7786f5b9d: Status 404 returned error can't find the container with id d8db89a27b7e4450000c8a669ca37cc356b1a2ff5398319fab8b33b7786f5b9d Jul 06 11:43:16.054762 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:16.054731 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b66961a3-8287-4d22-8198-c44fd2769504" path="/var/lib/kubelet/pods/b66961a3-8287-4d22-8198-c44fd2769504/volumes" Jul 06 11:43:16.141063 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:16.141031 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fc875b447-dlqtv" event={"ID":"27876b93-9b29-42c7-b21e-902222ff3daa","Type":"ContainerStarted","Data":"fb70863cf6a012948d2250479019f1d270b4a0b4141b8cc41986d2dffab09bb8"} Jul 06 11:43:16.141226 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:16.141091 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fc875b447-dlqtv" event={"ID":"27876b93-9b29-42c7-b21e-902222ff3daa","Type":"ContainerStarted","Data":"d8db89a27b7e4450000c8a669ca37cc356b1a2ff5398319fab8b33b7786f5b9d"} Jul 06 11:43:16.141226 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:16.141210 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5fc875b447-dlqtv" Jul 06 11:43:16.164708 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:16.164636 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5fc875b447-dlqtv" podStartSLOduration=1.589909411 podStartE2EDuration="2.16462041s" totalImagesPullingTime="574.710999ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:43:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:43:16.161982739 +0000 UTC m=+894.641781757" watchObservedRunningTime="2026-07-06 11:43:16.16462041 +0000 UTC m=+894.644419426" Jul 06 11:43:27.160808 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:43:27.160776 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5fc875b447-dlqtv" Jul 06 11:45:00.154940 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:45:00.154904 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722305-dwsj4"] Jul 06 11:45:00.158979 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:45:00.158950 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722305-dwsj4" Jul 06 11:45:00.185788 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:45:00.185755 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722305-dwsj4"] Jul 06 11:45:00.291185 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:45:00.291151 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2vwg\" (UniqueName: \"kubernetes.io/projected/a5604901-4af8-41b7-be0e-92cc18b812b1-kube-api-access-v2vwg\") pod \"maas-api-key-cleanup-29722305-dwsj4\" (UID: \"a5604901-4af8-41b7-be0e-92cc18b812b1\") " pod="opendatahub/maas-api-key-cleanup-29722305-dwsj4" Jul 06 11:45:00.392484 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:45:00.392446 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2vwg\" (UniqueName: \"kubernetes.io/projected/a5604901-4af8-41b7-be0e-92cc18b812b1-kube-api-access-v2vwg\") pod \"maas-api-key-cleanup-29722305-dwsj4\" (UID: \"a5604901-4af8-41b7-be0e-92cc18b812b1\") " pod="opendatahub/maas-api-key-cleanup-29722305-dwsj4" Jul 06 11:45:00.409293 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:45:00.409211 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2vwg\" (UniqueName: \"kubernetes.io/projected/a5604901-4af8-41b7-be0e-92cc18b812b1-kube-api-access-v2vwg\") pod \"maas-api-key-cleanup-29722305-dwsj4\" (UID: \"a5604901-4af8-41b7-be0e-92cc18b812b1\") " pod="opendatahub/maas-api-key-cleanup-29722305-dwsj4" Jul 06 11:45:00.468900 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:45:00.468863 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722305-dwsj4" Jul 06 11:45:00.630218 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:45:00.630190 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722305-dwsj4"] Jul 06 11:45:00.632769 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:45:00.632732 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5604901_4af8_41b7_be0e_92cc18b812b1.slice/crio-5997b08194e7b8ada8ed6f0666b7c03688b3cf23a40c34c0d4e4f19f71621fa6 WatchSource:0}: Error finding container 5997b08194e7b8ada8ed6f0666b7c03688b3cf23a40c34c0d4e4f19f71621fa6: Status 404 returned error can't find the container with id 5997b08194e7b8ada8ed6f0666b7c03688b3cf23a40c34c0d4e4f19f71621fa6 Jul 06 11:45:00.634958 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:45:00.634937 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 11:45:00.727461 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:45:00.727371 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722305-dwsj4" event={"ID":"a5604901-4af8-41b7-be0e-92cc18b812b1","Type":"ContainerStarted","Data":"5997b08194e7b8ada8ed6f0666b7c03688b3cf23a40c34c0d4e4f19f71621fa6"} Jul 06 11:45:03.748361 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:45:03.748323 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722305-dwsj4" event={"ID":"a5604901-4af8-41b7-be0e-92cc18b812b1","Type":"ContainerStarted","Data":"abd1ab8d8776129551b93eb41080dfa5f3809027b478c922c44892d46d480aad"} Jul 06 11:45:03.774945 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:45:03.774888 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29722305-dwsj4" podStartSLOduration=1.572006679 podStartE2EDuration="3.774871205s" totalImagesPullingTime="2.202864526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:45:03.773827013 +0000 UTC m=+1002.253626030" watchObservedRunningTime="2026-07-06 11:45:03.774871205 +0000 UTC m=+1002.254670222" Jul 06 11:45:04.755066 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:45:04.755031 2568 generic.go:356] "Generic (PLEG): container finished" podID="a5604901-4af8-41b7-be0e-92cc18b812b1" containerID="abd1ab8d8776129551b93eb41080dfa5f3809027b478c922c44892d46d480aad" exitCode=0 Jul 06 11:45:04.755457 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:45:04.755110 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722305-dwsj4" event={"ID":"a5604901-4af8-41b7-be0e-92cc18b812b1","Type":"ContainerDied","Data":"abd1ab8d8776129551b93eb41080dfa5f3809027b478c922c44892d46d480aad"} Jul 06 11:45:05.901784 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:45:05.901755 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722305-dwsj4" Jul 06 11:45:06.048039 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:45:06.047944 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2vwg\" (UniqueName: \"kubernetes.io/projected/a5604901-4af8-41b7-be0e-92cc18b812b1-kube-api-access-v2vwg\") pod \"a5604901-4af8-41b7-be0e-92cc18b812b1\" (UID: \"a5604901-4af8-41b7-be0e-92cc18b812b1\") " Jul 06 11:45:06.050223 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:45:06.050192 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5604901-4af8-41b7-be0e-92cc18b812b1-kube-api-access-v2vwg" (OuterVolumeSpecName: "kube-api-access-v2vwg") pod "a5604901-4af8-41b7-be0e-92cc18b812b1" (UID: "a5604901-4af8-41b7-be0e-92cc18b812b1"). InnerVolumeSpecName "kube-api-access-v2vwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 11:45:06.149629 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:45:06.149596 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2vwg\" (UniqueName: \"kubernetes.io/projected/a5604901-4af8-41b7-be0e-92cc18b812b1-kube-api-access-v2vwg\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:45:06.767067 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:45:06.767032 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722305-dwsj4" event={"ID":"a5604901-4af8-41b7-be0e-92cc18b812b1","Type":"ContainerDied","Data":"5997b08194e7b8ada8ed6f0666b7c03688b3cf23a40c34c0d4e4f19f71621fa6"} Jul 06 11:45:06.767067 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:45:06.767063 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722305-dwsj4" Jul 06 11:45:06.767293 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:45:06.767087 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5997b08194e7b8ada8ed6f0666b7c03688b3cf23a40c34c0d4e4f19f71621fa6" Jul 06 11:48:17.513033 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:48:17.512998 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9k4gr"] Jul 06 11:48:17.513590 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:48:17.513221 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9k4gr" podUID="61b231f5-3f7a-4194-bb88-aedf63314d5f" containerName="manager" containerID="cri-o://ac83a0d689724ef53cc793e42af5d02c1ab5ec3d233aed77503aaac393829b05" gracePeriod=10 Jul 06 11:48:20.674732 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:48:20.674700 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9k4gr" Jul 06 11:48:20.782158 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:48:20.782061 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9tpg\" (UniqueName: \"kubernetes.io/projected/61b231f5-3f7a-4194-bb88-aedf63314d5f-kube-api-access-k9tpg\") pod \"61b231f5-3f7a-4194-bb88-aedf63314d5f\" (UID: \"61b231f5-3f7a-4194-bb88-aedf63314d5f\") " Jul 06 11:48:20.782158 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:48:20.782128 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61b231f5-3f7a-4194-bb88-aedf63314d5f-extensions-socket-volume\") pod \"61b231f5-3f7a-4194-bb88-aedf63314d5f\" (UID: \"61b231f5-3f7a-4194-bb88-aedf63314d5f\") " Jul 06 11:48:20.782500 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:48:20.782476 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b231f5-3f7a-4194-bb88-aedf63314d5f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "61b231f5-3f7a-4194-bb88-aedf63314d5f" (UID: "61b231f5-3f7a-4194-bb88-aedf63314d5f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:48:20.784160 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:48:20.784138 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b231f5-3f7a-4194-bb88-aedf63314d5f-kube-api-access-k9tpg" (OuterVolumeSpecName: "kube-api-access-k9tpg") pod "61b231f5-3f7a-4194-bb88-aedf63314d5f" (UID: "61b231f5-3f7a-4194-bb88-aedf63314d5f"). InnerVolumeSpecName "kube-api-access-k9tpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 11:48:20.807547 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:48:20.807503 2568 generic.go:356] "Generic (PLEG): container finished" podID="61b231f5-3f7a-4194-bb88-aedf63314d5f" containerID="ac83a0d689724ef53cc793e42af5d02c1ab5ec3d233aed77503aaac393829b05" exitCode=0 Jul 06 11:48:20.807727 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:48:20.807554 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9k4gr" Jul 06 11:48:20.807727 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:48:20.807579 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9k4gr" event={"ID":"61b231f5-3f7a-4194-bb88-aedf63314d5f","Type":"ContainerDied","Data":"ac83a0d689724ef53cc793e42af5d02c1ab5ec3d233aed77503aaac393829b05"} Jul 06 11:48:20.807727 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:48:20.807643 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9k4gr" event={"ID":"61b231f5-3f7a-4194-bb88-aedf63314d5f","Type":"ContainerDied","Data":"bb65e139786c178b0055f58d220d8db6f31d3171127adc16a932acb3ecda6fce"} Jul 06 11:48:20.807727 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:48:20.807671 2568 scope.go:117] "RemoveContainer" containerID="ac83a0d689724ef53cc793e42af5d02c1ab5ec3d233aed77503aaac393829b05" Jul 06 11:48:20.818577 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:48:20.818553 2568 scope.go:117] "RemoveContainer" containerID="ac83a0d689724ef53cc793e42af5d02c1ab5ec3d233aed77503aaac393829b05" Jul 06 11:48:20.818913 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:48:20.818895 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac83a0d689724ef53cc793e42af5d02c1ab5ec3d233aed77503aaac393829b05\": container with ID starting with ac83a0d689724ef53cc793e42af5d02c1ab5ec3d233aed77503aaac393829b05 not found: ID does not exist" containerID="ac83a0d689724ef53cc793e42af5d02c1ab5ec3d233aed77503aaac393829b05" Jul 06 11:48:20.818983 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:48:20.818922 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac83a0d689724ef53cc793e42af5d02c1ab5ec3d233aed77503aaac393829b05"} err="failed to get container status \"ac83a0d689724ef53cc793e42af5d02c1ab5ec3d233aed77503aaac393829b05\": rpc error: code = NotFound desc = could not find container \"ac83a0d689724ef53cc793e42af5d02c1ab5ec3d233aed77503aaac393829b05\": container with ID starting with ac83a0d689724ef53cc793e42af5d02c1ab5ec3d233aed77503aaac393829b05 not found: ID does not exist" Jul 06 11:48:20.843567 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:48:20.843540 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9k4gr"] Jul 06 11:48:20.845693 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:48:20.845659 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9k4gr"] Jul 06 11:48:20.883298 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:48:20.883263 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9tpg\" (UniqueName: \"kubernetes.io/projected/61b231f5-3f7a-4194-bb88-aedf63314d5f-kube-api-access-k9tpg\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:48:20.883298 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:48:20.883292 2568 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61b231f5-3f7a-4194-bb88-aedf63314d5f-extensions-socket-volume\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:48:22.053694 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:48:22.053644 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61b231f5-3f7a-4194-bb88-aedf63314d5f" path="/var/lib/kubelet/pods/61b231f5-3f7a-4194-bb88-aedf63314d5f/volumes" Jul 06 11:49:18.712792 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:49:18.712749 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vtfn2"] Jul 06 11:49:18.713343 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:49:18.713103 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61b231f5-3f7a-4194-bb88-aedf63314d5f" containerName="manager" Jul 06 11:49:18.713343 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:49:18.713115 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="61b231f5-3f7a-4194-bb88-aedf63314d5f" containerName="manager" Jul 06 11:49:18.713343 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:49:18.713125 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5604901-4af8-41b7-be0e-92cc18b812b1" containerName="cleanup" Jul 06 11:49:18.713343 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:49:18.713131 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5604901-4af8-41b7-be0e-92cc18b812b1" containerName="cleanup" Jul 06 11:49:18.713343 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:49:18.713198 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="61b231f5-3f7a-4194-bb88-aedf63314d5f" containerName="manager" Jul 06 11:49:18.713343 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:49:18.713205 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5604901-4af8-41b7-be0e-92cc18b812b1" containerName="cleanup" Jul 06 11:49:18.716421 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:49:18.716398 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vtfn2" Jul 06 11:49:18.719833 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:49:18.719807 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qlxfr\"" Jul 06 11:49:18.733403 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:49:18.733376 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vtfn2"] Jul 06 11:49:18.818063 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:49:18.818028 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cswg6\" (UniqueName: \"kubernetes.io/projected/63b1cd37-c490-410b-92bd-be6d823a4efe-kube-api-access-cswg6\") pod \"kuadrant-operator-controller-manager-86878c769f-vtfn2\" (UID: \"63b1cd37-c490-410b-92bd-be6d823a4efe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vtfn2" Jul 06 11:49:18.818263 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:49:18.818116 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/63b1cd37-c490-410b-92bd-be6d823a4efe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vtfn2\" (UID: \"63b1cd37-c490-410b-92bd-be6d823a4efe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vtfn2" Jul 06 11:49:18.918855 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:49:18.918811 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/63b1cd37-c490-410b-92bd-be6d823a4efe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vtfn2\" (UID: \"63b1cd37-c490-410b-92bd-be6d823a4efe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vtfn2" Jul 06 11:49:18.918855 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:49:18.918863 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cswg6\" (UniqueName: \"kubernetes.io/projected/63b1cd37-c490-410b-92bd-be6d823a4efe-kube-api-access-cswg6\") pod \"kuadrant-operator-controller-manager-86878c769f-vtfn2\" (UID: \"63b1cd37-c490-410b-92bd-be6d823a4efe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vtfn2" Jul 06 11:49:18.919215 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:49:18.919194 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/63b1cd37-c490-410b-92bd-be6d823a4efe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vtfn2\" (UID: \"63b1cd37-c490-410b-92bd-be6d823a4efe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vtfn2" Jul 06 11:49:18.933387 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:49:18.933362 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cswg6\" (UniqueName: \"kubernetes.io/projected/63b1cd37-c490-410b-92bd-be6d823a4efe-kube-api-access-cswg6\") pod \"kuadrant-operator-controller-manager-86878c769f-vtfn2\" (UID: \"63b1cd37-c490-410b-92bd-be6d823a4efe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vtfn2" Jul 06 11:49:19.027096 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:49:19.027003 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vtfn2" Jul 06 11:49:19.186407 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:49:19.186376 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vtfn2"] Jul 06 11:49:19.187332 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:49:19.187308 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b1cd37_c490_410b_92bd_be6d823a4efe.slice/crio-1fadba6d23ffb5a3cc3b93fbe953f75b1efdec1f0145812ad1a2281dfbe36fae WatchSource:0}: Error finding container 1fadba6d23ffb5a3cc3b93fbe953f75b1efdec1f0145812ad1a2281dfbe36fae: Status 404 returned error can't find the container with id 1fadba6d23ffb5a3cc3b93fbe953f75b1efdec1f0145812ad1a2281dfbe36fae Jul 06 11:49:20.119396 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:49:20.119354 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vtfn2" event={"ID":"63b1cd37-c490-410b-92bd-be6d823a4efe","Type":"ContainerStarted","Data":"3e49de7be743a1e28129c20f41a09495b1d611a100d54e0937242929f4f5f594"} Jul 06 11:49:20.119860 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:49:20.119412 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vtfn2" event={"ID":"63b1cd37-c490-410b-92bd-be6d823a4efe","Type":"ContainerStarted","Data":"1fadba6d23ffb5a3cc3b93fbe953f75b1efdec1f0145812ad1a2281dfbe36fae"} Jul 06 11:49:20.119860 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:49:20.119493 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vtfn2" Jul 06 11:49:20.152885 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:49:20.152796 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vtfn2" podStartSLOduration=2.152779962 podStartE2EDuration="2.152779962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:49:20.151465725 +0000 UTC m=+1258.631264771" watchObservedRunningTime="2026-07-06 11:49:20.152779962 +0000 UTC m=+1258.632578979" Jul 06 11:49:31.127304 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:49:31.127268 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vtfn2" Jul 06 11:55:13.154937 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.154893 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx"] Jul 06 11:55:13.158698 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.158654 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:13.161811 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.161793 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8025cb00-openshift-default-dockercfg-nkk8w\"" Jul 06 11:55:13.179423 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.179403 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx"] Jul 06 11:55:13.254312 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.254278 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7762871f-1d49-4017-bf75-e811c2a691d0-istio-token\") pod \"e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:13.254312 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.254314 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7762871f-1d49-4017-bf75-e811c2a691d0-workload-socket\") pod \"e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:13.254518 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.254343 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2tp9\" (UniqueName: \"kubernetes.io/projected/7762871f-1d49-4017-bf75-e811c2a691d0-kube-api-access-c2tp9\") pod \"e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:13.254518 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.254422 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7762871f-1d49-4017-bf75-e811c2a691d0-istio-envoy\") pod \"e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:13.254518 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.254447 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7762871f-1d49-4017-bf75-e811c2a691d0-istio-podinfo\") pod \"e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:13.254518 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.254473 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7762871f-1d49-4017-bf75-e811c2a691d0-istiod-ca-cert\") pod \"e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:13.254650 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.254531 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7762871f-1d49-4017-bf75-e811c2a691d0-credential-socket\") pod \"e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:13.254650 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.254549 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7762871f-1d49-4017-bf75-e811c2a691d0-istio-data\") pod \"e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:13.254650 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.254565 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7762871f-1d49-4017-bf75-e811c2a691d0-workload-certs\") pod \"e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:13.355623 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.355581 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7762871f-1d49-4017-bf75-e811c2a691d0-istio-token\") pod \"e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:13.355623 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.355625 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7762871f-1d49-4017-bf75-e811c2a691d0-workload-socket\") pod \"e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:13.355916 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.355758 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2tp9\" (UniqueName: \"kubernetes.io/projected/7762871f-1d49-4017-bf75-e811c2a691d0-kube-api-access-c2tp9\") pod \"e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:13.355916 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.355827 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7762871f-1d49-4017-bf75-e811c2a691d0-istio-envoy\") pod \"e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:13.355916 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.355851 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7762871f-1d49-4017-bf75-e811c2a691d0-istio-podinfo\") pod \"e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:13.355916 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.355881 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7762871f-1d49-4017-bf75-e811c2a691d0-istiod-ca-cert\") pod \"e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:13.356144 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.355956 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7762871f-1d49-4017-bf75-e811c2a691d0-credential-socket\") pod \"e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:13.356144 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.356000 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7762871f-1d49-4017-bf75-e811c2a691d0-istio-data\") pod \"e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:13.356144 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.356029 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7762871f-1d49-4017-bf75-e811c2a691d0-workload-certs\") pod \"e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:13.356144 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.356056 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7762871f-1d49-4017-bf75-e811c2a691d0-workload-socket\") pod \"e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:13.356348 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.356289 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7762871f-1d49-4017-bf75-e811c2a691d0-credential-socket\") pod \"e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:13.356348 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.356316 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7762871f-1d49-4017-bf75-e811c2a691d0-istio-data\") pod \"e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:13.356452 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.356400 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7762871f-1d49-4017-bf75-e811c2a691d0-workload-certs\") pod \"e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:13.356541 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.356518 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7762871f-1d49-4017-bf75-e811c2a691d0-istiod-ca-cert\") pod \"e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:13.358047 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.358018 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7762871f-1d49-4017-bf75-e811c2a691d0-istio-envoy\") pod \"e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:13.358252 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.358231 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7762871f-1d49-4017-bf75-e811c2a691d0-istio-podinfo\") pod \"e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:13.364087 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.364066 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7762871f-1d49-4017-bf75-e811c2a691d0-istio-token\") pod \"e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:13.364297 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.364277 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2tp9\" (UniqueName: \"kubernetes.io/projected/7762871f-1d49-4017-bf75-e811c2a691d0-kube-api-access-c2tp9\") pod \"e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:13.469281 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.469247 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:13.606151 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.606122 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx"] Jul 06 11:55:13.607534 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:55:13.607496 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7762871f_1d49_4017_bf75_e811c2a691d0.slice/crio-d35819d7e90e9d1edd79ad9987ee83f5c20d749436eff61e13c0cd9c457deb9c WatchSource:0}: Error finding container d35819d7e90e9d1edd79ad9987ee83f5c20d749436eff61e13c0cd9c457deb9c: Status 404 returned error can't find the container with id d35819d7e90e9d1edd79ad9987ee83f5c20d749436eff61e13c0cd9c457deb9c Jul 06 11:55:13.609581 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.609547 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 11:55:13.609652 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.609617 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 11:55:13.609652 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.609643 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 11:55:13.975319 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.975282 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" event={"ID":"7762871f-1d49-4017-bf75-e811c2a691d0","Type":"ContainerStarted","Data":"0bd39b7f7b4496465a78ef9b76f62612b59ce6a5d9010642d91879a1f1fdacac"} Jul 06 11:55:13.975490 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:13.975334 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" event={"ID":"7762871f-1d49-4017-bf75-e811c2a691d0","Type":"ContainerStarted","Data":"d35819d7e90e9d1edd79ad9987ee83f5c20d749436eff61e13c0cd9c457deb9c"} Jul 06 11:55:14.373169 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:14.373040 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" podStartSLOduration=1.372987008 podStartE2EDuration="1.372987008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:55:13.99559792 +0000 UTC m=+1612.475396936" watchObservedRunningTime="2026-07-06 11:55:14.372987008 +0000 UTC m=+1612.852786026" Jul 06 11:55:14.374044 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:14.374021 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg"] Jul 06 11:55:14.378425 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:14.378402 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg" Jul 06 11:55:14.380992 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:14.380975 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-8025cb00-serving-cert\"" Jul 06 11:55:14.385121 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:14.385099 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg"] Jul 06 11:55:14.466836 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:14.466805 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37afc2bd-bc70-4817-9b42-ead24207664f-openshift-service-ca\") pod \"maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg\" (UID: \"37afc2bd-bc70-4817-9b42-ead24207664f\") " pod="opendatahub/maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg" Jul 06 11:55:14.466988 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:14.466853 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7ztc\" (UniqueName: \"kubernetes.io/projected/37afc2bd-bc70-4817-9b42-ead24207664f-kube-api-access-c7ztc\") pod \"maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg\" (UID: \"37afc2bd-bc70-4817-9b42-ead24207664f\") " pod="opendatahub/maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg" Jul 06 11:55:14.466988 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:14.466883 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37afc2bd-bc70-4817-9b42-ead24207664f-maas-api-tls\") pod \"maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg\" (UID: \"37afc2bd-bc70-4817-9b42-ead24207664f\") " pod="opendatahub/maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg" Jul 06 11:55:14.469731 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:14.469711 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:14.490721 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:14.490670 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:14.567603 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:14.567573 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37afc2bd-bc70-4817-9b42-ead24207664f-openshift-service-ca\") pod \"maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg\" (UID: \"37afc2bd-bc70-4817-9b42-ead24207664f\") " pod="opendatahub/maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg" Jul 06 11:55:14.567790 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:14.567624 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7ztc\" (UniqueName: \"kubernetes.io/projected/37afc2bd-bc70-4817-9b42-ead24207664f-kube-api-access-c7ztc\") pod \"maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg\" (UID: \"37afc2bd-bc70-4817-9b42-ead24207664f\") " pod="opendatahub/maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg" Jul 06 11:55:14.567790 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:14.567698 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37afc2bd-bc70-4817-9b42-ead24207664f-maas-api-tls\") pod \"maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg\" (UID: \"37afc2bd-bc70-4817-9b42-ead24207664f\") " pod="opendatahub/maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg" Jul 06 11:55:14.567877 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:55:14.567855 2568 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-8025cb00-serving-cert: secret "maas-api-e2e-ait-8025cb00-serving-cert" not found Jul 06 11:55:14.567938 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:55:14.567928 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37afc2bd-bc70-4817-9b42-ead24207664f-maas-api-tls podName:37afc2bd-bc70-4817-9b42-ead24207664f nodeName:}" failed. No retries permitted until 2026-07-06 11:55:15.067907426 +0000 UTC m=+1613.547706433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/37afc2bd-bc70-4817-9b42-ead24207664f-maas-api-tls") pod "maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg" (UID: "37afc2bd-bc70-4817-9b42-ead24207664f") : secret "maas-api-e2e-ait-8025cb00-serving-cert" not found Jul 06 11:55:14.568248 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:14.568230 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37afc2bd-bc70-4817-9b42-ead24207664f-openshift-service-ca\") pod \"maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg\" (UID: \"37afc2bd-bc70-4817-9b42-ead24207664f\") " pod="opendatahub/maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg" Jul 06 11:55:14.583281 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:14.583254 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7ztc\" (UniqueName: \"kubernetes.io/projected/37afc2bd-bc70-4817-9b42-ead24207664f-kube-api-access-c7ztc\") pod \"maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg\" (UID: \"37afc2bd-bc70-4817-9b42-ead24207664f\") " pod="opendatahub/maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg" Jul 06 11:55:14.981000 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:14.980962 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:14.981803 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:14.981782 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:15.072722 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:15.072426 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37afc2bd-bc70-4817-9b42-ead24207664f-maas-api-tls\") pod \"maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg\" (UID: \"37afc2bd-bc70-4817-9b42-ead24207664f\") " pod="opendatahub/maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg" Jul 06 11:55:15.074982 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:15.074957 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37afc2bd-bc70-4817-9b42-ead24207664f-maas-api-tls\") pod \"maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg\" (UID: \"37afc2bd-bc70-4817-9b42-ead24207664f\") " pod="opendatahub/maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg" Jul 06 11:55:15.290483 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:15.290404 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg" Jul 06 11:55:15.427610 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:15.427588 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg"] Jul 06 11:55:15.429902 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:55:15.429802 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37afc2bd_bc70_4817_9b42_ead24207664f.slice/crio-cc92bad68f29262377113e310faf51e050301cab94a376154a090d3262853897 WatchSource:0}: Error finding container cc92bad68f29262377113e310faf51e050301cab94a376154a090d3262853897: Status 404 returned error can't find the container with id cc92bad68f29262377113e310faf51e050301cab94a376154a090d3262853897 Jul 06 11:55:15.431716 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:15.431657 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 11:55:15.987078 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:15.987045 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg" event={"ID":"37afc2bd-bc70-4817-9b42-ead24207664f","Type":"ContainerStarted","Data":"f2be06807a2991a31ebe7581b72e3b209d17585c9de6089e88588486e9f751f8"} Jul 06 11:55:15.987197 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:15.987106 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg" event={"ID":"37afc2bd-bc70-4817-9b42-ead24207664f","Type":"ContainerStarted","Data":"cc92bad68f29262377113e310faf51e050301cab94a376154a090d3262853897"} Jul 06 11:55:15.988596 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:15.988570 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg" Jul 06 11:55:16.011298 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:16.011258 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg" podStartSLOduration=1.553213564 podStartE2EDuration="2.011244836s" totalImagesPullingTime="458.031272ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:55:16.009796758 +0000 UTC m=+1614.489595774" watchObservedRunningTime="2026-07-06 11:55:16.011244836 +0000 UTC m=+1614.491043861" Jul 06 11:55:19.586989 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:19.586955 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx"] Jul 06 11:55:19.587443 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:19.587130 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" podUID="7762871f-1d49-4017-bf75-e811c2a691d0" containerName="istio-proxy" containerID="cri-o://0bd39b7f7b4496465a78ef9b76f62612b59ce6a5d9010642d91879a1f1fdacac" gracePeriod=30 Jul 06 11:55:19.843044 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:19.842252 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg"] Jul 06 11:55:19.843044 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:19.842471 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg" podUID="37afc2bd-bc70-4817-9b42-ead24207664f" containerName="maas-api" containerID="cri-o://f2be06807a2991a31ebe7581b72e3b209d17585c9de6089e88588486e9f751f8" gracePeriod=30 Jul 06 11:55:20.010242 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:20.010212 2568 generic.go:356] "Generic (PLEG): container finished" podID="37afc2bd-bc70-4817-9b42-ead24207664f" containerID="f2be06807a2991a31ebe7581b72e3b209d17585c9de6089e88588486e9f751f8" exitCode=0 Jul 06 11:55:20.010405 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:20.010287 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg" event={"ID":"37afc2bd-bc70-4817-9b42-ead24207664f","Type":"ContainerDied","Data":"f2be06807a2991a31ebe7581b72e3b209d17585c9de6089e88588486e9f751f8"} Jul 06 11:55:20.115322 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:20.115293 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg" Jul 06 11:55:20.223421 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:20.223387 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37afc2bd-bc70-4817-9b42-ead24207664f-openshift-service-ca\") pod \"37afc2bd-bc70-4817-9b42-ead24207664f\" (UID: \"37afc2bd-bc70-4817-9b42-ead24207664f\") " Jul 06 11:55:20.223568 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:20.223468 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37afc2bd-bc70-4817-9b42-ead24207664f-maas-api-tls\") pod \"37afc2bd-bc70-4817-9b42-ead24207664f\" (UID: \"37afc2bd-bc70-4817-9b42-ead24207664f\") " Jul 06 11:55:20.223568 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:20.223500 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7ztc\" (UniqueName: \"kubernetes.io/projected/37afc2bd-bc70-4817-9b42-ead24207664f-kube-api-access-c7ztc\") pod \"37afc2bd-bc70-4817-9b42-ead24207664f\" (UID: \"37afc2bd-bc70-4817-9b42-ead24207664f\") " Jul 06 11:55:20.223775 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:20.223752 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37afc2bd-bc70-4817-9b42-ead24207664f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "37afc2bd-bc70-4817-9b42-ead24207664f" (UID: "37afc2bd-bc70-4817-9b42-ead24207664f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 11:55:20.225484 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:20.225452 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37afc2bd-bc70-4817-9b42-ead24207664f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "37afc2bd-bc70-4817-9b42-ead24207664f" (UID: "37afc2bd-bc70-4817-9b42-ead24207664f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 11:55:20.225594 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:20.225489 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37afc2bd-bc70-4817-9b42-ead24207664f-kube-api-access-c7ztc" (OuterVolumeSpecName: "kube-api-access-c7ztc") pod "37afc2bd-bc70-4817-9b42-ead24207664f" (UID: "37afc2bd-bc70-4817-9b42-ead24207664f"). InnerVolumeSpecName "kube-api-access-c7ztc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 11:55:20.324723 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:20.324662 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37afc2bd-bc70-4817-9b42-ead24207664f-openshift-service-ca\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:20.324723 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:20.324719 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37afc2bd-bc70-4817-9b42-ead24207664f-maas-api-tls\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:20.324723 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:20.324729 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7ztc\" (UniqueName: \"kubernetes.io/projected/37afc2bd-bc70-4817-9b42-ead24207664f-kube-api-access-c7ztc\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:21.017481 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:21.017444 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg" event={"ID":"37afc2bd-bc70-4817-9b42-ead24207664f","Type":"ContainerDied","Data":"cc92bad68f29262377113e310faf51e050301cab94a376154a090d3262853897"} Jul 06 11:55:21.017961 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:21.017508 2568 scope.go:117] "RemoveContainer" containerID="f2be06807a2991a31ebe7581b72e3b209d17585c9de6089e88588486e9f751f8" Jul 06 11:55:21.017961 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:21.017713 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg" Jul 06 11:55:21.051441 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:21.051403 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg"] Jul 06 11:55:21.064883 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:21.064595 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-8025cb00-695d6f64f9-mjwpg"] Jul 06 11:55:22.053292 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:22.053260 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37afc2bd-bc70-4817-9b42-ead24207664f" path="/var/lib/kubelet/pods/37afc2bd-bc70-4817-9b42-ead24207664f/volumes" Jul 06 11:55:24.933134 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:24.933110 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:25.042301 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.042211 2568 generic.go:356] "Generic (PLEG): container finished" podID="7762871f-1d49-4017-bf75-e811c2a691d0" containerID="0bd39b7f7b4496465a78ef9b76f62612b59ce6a5d9010642d91879a1f1fdacac" exitCode=0 Jul 06 11:55:25.042301 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.042258 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" Jul 06 11:55:25.042486 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.042298 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" event={"ID":"7762871f-1d49-4017-bf75-e811c2a691d0","Type":"ContainerDied","Data":"0bd39b7f7b4496465a78ef9b76f62612b59ce6a5d9010642d91879a1f1fdacac"} Jul 06 11:55:25.042486 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.042355 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx" event={"ID":"7762871f-1d49-4017-bf75-e811c2a691d0","Type":"ContainerDied","Data":"d35819d7e90e9d1edd79ad9987ee83f5c20d749436eff61e13c0cd9c457deb9c"} Jul 06 11:55:25.042486 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.042378 2568 scope.go:117] "RemoveContainer" containerID="0bd39b7f7b4496465a78ef9b76f62612b59ce6a5d9010642d91879a1f1fdacac" Jul 06 11:55:25.051191 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.051177 2568 scope.go:117] "RemoveContainer" containerID="0bd39b7f7b4496465a78ef9b76f62612b59ce6a5d9010642d91879a1f1fdacac" Jul 06 11:55:25.051424 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:55:25.051408 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bd39b7f7b4496465a78ef9b76f62612b59ce6a5d9010642d91879a1f1fdacac\": container with ID starting with 0bd39b7f7b4496465a78ef9b76f62612b59ce6a5d9010642d91879a1f1fdacac not found: ID does not exist" containerID="0bd39b7f7b4496465a78ef9b76f62612b59ce6a5d9010642d91879a1f1fdacac" Jul 06 11:55:25.051463 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.051434 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bd39b7f7b4496465a78ef9b76f62612b59ce6a5d9010642d91879a1f1fdacac"} err="failed to get container status \"0bd39b7f7b4496465a78ef9b76f62612b59ce6a5d9010642d91879a1f1fdacac\": rpc error: code = NotFound desc = could not find container \"0bd39b7f7b4496465a78ef9b76f62612b59ce6a5d9010642d91879a1f1fdacac\": container with ID starting with 0bd39b7f7b4496465a78ef9b76f62612b59ce6a5d9010642d91879a1f1fdacac not found: ID does not exist" Jul 06 11:55:25.075762 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.075739 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7762871f-1d49-4017-bf75-e811c2a691d0-workload-certs\") pod \"7762871f-1d49-4017-bf75-e811c2a691d0\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " Jul 06 11:55:25.075867 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.075783 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7762871f-1d49-4017-bf75-e811c2a691d0-istio-podinfo\") pod \"7762871f-1d49-4017-bf75-e811c2a691d0\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " Jul 06 11:55:25.075867 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.075798 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7762871f-1d49-4017-bf75-e811c2a691d0-istio-data\") pod \"7762871f-1d49-4017-bf75-e811c2a691d0\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " Jul 06 11:55:25.075867 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.075838 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7762871f-1d49-4017-bf75-e811c2a691d0-istio-token\") pod \"7762871f-1d49-4017-bf75-e811c2a691d0\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " Jul 06 11:55:25.075867 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.075854 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2tp9\" (UniqueName: \"kubernetes.io/projected/7762871f-1d49-4017-bf75-e811c2a691d0-kube-api-access-c2tp9\") pod \"7762871f-1d49-4017-bf75-e811c2a691d0\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " Jul 06 11:55:25.076078 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.075875 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7762871f-1d49-4017-bf75-e811c2a691d0-istiod-ca-cert\") pod \"7762871f-1d49-4017-bf75-e811c2a691d0\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " Jul 06 11:55:25.076078 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.075932 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7762871f-1d49-4017-bf75-e811c2a691d0-workload-socket\") pod \"7762871f-1d49-4017-bf75-e811c2a691d0\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " Jul 06 11:55:25.076078 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.075958 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7762871f-1d49-4017-bf75-e811c2a691d0-istio-envoy\") pod \"7762871f-1d49-4017-bf75-e811c2a691d0\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " Jul 06 11:55:25.076078 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.075981 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7762871f-1d49-4017-bf75-e811c2a691d0-credential-socket\") pod \"7762871f-1d49-4017-bf75-e811c2a691d0\" (UID: \"7762871f-1d49-4017-bf75-e811c2a691d0\") " Jul 06 11:55:25.076078 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.076005 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7762871f-1d49-4017-bf75-e811c2a691d0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7762871f-1d49-4017-bf75-e811c2a691d0" (UID: "7762871f-1d49-4017-bf75-e811c2a691d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:55:25.076340 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.076231 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7762871f-1d49-4017-bf75-e811c2a691d0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7762871f-1d49-4017-bf75-e811c2a691d0" (UID: "7762871f-1d49-4017-bf75-e811c2a691d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:55:25.076340 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.076255 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7762871f-1d49-4017-bf75-e811c2a691d0-workload-certs\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:25.076467 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.076445 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7762871f-1d49-4017-bf75-e811c2a691d0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7762871f-1d49-4017-bf75-e811c2a691d0" (UID: "7762871f-1d49-4017-bf75-e811c2a691d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:55:25.076525 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.076474 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7762871f-1d49-4017-bf75-e811c2a691d0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7762871f-1d49-4017-bf75-e811c2a691d0" (UID: "7762871f-1d49-4017-bf75-e811c2a691d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 11:55:25.076749 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.076723 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7762871f-1d49-4017-bf75-e811c2a691d0-istio-data" (OuterVolumeSpecName: "istio-data") pod "7762871f-1d49-4017-bf75-e811c2a691d0" (UID: "7762871f-1d49-4017-bf75-e811c2a691d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:55:25.078122 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.078095 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7762871f-1d49-4017-bf75-e811c2a691d0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7762871f-1d49-4017-bf75-e811c2a691d0" (UID: "7762871f-1d49-4017-bf75-e811c2a691d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 11:55:25.078219 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.078147 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7762871f-1d49-4017-bf75-e811c2a691d0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7762871f-1d49-4017-bf75-e811c2a691d0" (UID: "7762871f-1d49-4017-bf75-e811c2a691d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:55:25.078219 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.078199 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7762871f-1d49-4017-bf75-e811c2a691d0-kube-api-access-c2tp9" (OuterVolumeSpecName: "kube-api-access-c2tp9") pod "7762871f-1d49-4017-bf75-e811c2a691d0" (UID: "7762871f-1d49-4017-bf75-e811c2a691d0"). InnerVolumeSpecName "kube-api-access-c2tp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 11:55:25.078295 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.078206 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7762871f-1d49-4017-bf75-e811c2a691d0-istio-token" (OuterVolumeSpecName: "istio-token") pod "7762871f-1d49-4017-bf75-e811c2a691d0" (UID: "7762871f-1d49-4017-bf75-e811c2a691d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 11:55:25.177334 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.177292 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7762871f-1d49-4017-bf75-e811c2a691d0-istio-token\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:25.177334 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.177333 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2tp9\" (UniqueName: \"kubernetes.io/projected/7762871f-1d49-4017-bf75-e811c2a691d0-kube-api-access-c2tp9\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:25.177539 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.177351 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7762871f-1d49-4017-bf75-e811c2a691d0-istiod-ca-cert\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:25.177539 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.177366 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7762871f-1d49-4017-bf75-e811c2a691d0-workload-socket\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:25.177539 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.177380 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7762871f-1d49-4017-bf75-e811c2a691d0-istio-envoy\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:25.177539 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.177393 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7762871f-1d49-4017-bf75-e811c2a691d0-credential-socket\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:25.177539 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.177405 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7762871f-1d49-4017-bf75-e811c2a691d0-istio-podinfo\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:25.177539 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.177415 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7762871f-1d49-4017-bf75-e811c2a691d0-istio-data\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:25.366435 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.366404 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx"] Jul 06 11:55:25.370167 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.370143 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8025cb00-openshift-default-8585df54d7-f9hlx"] Jul 06 11:55:25.871209 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.871164 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz"] Jul 06 11:55:25.871962 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.871936 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37afc2bd-bc70-4817-9b42-ead24207664f" containerName="maas-api" Jul 06 11:55:25.872134 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.872118 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="37afc2bd-bc70-4817-9b42-ead24207664f" containerName="maas-api" Jul 06 11:55:25.872315 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.872289 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7762871f-1d49-4017-bf75-e811c2a691d0" containerName="istio-proxy" Jul 06 11:55:25.872315 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.872309 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7762871f-1d49-4017-bf75-e811c2a691d0" containerName="istio-proxy" Jul 06 11:55:25.872535 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.872446 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="37afc2bd-bc70-4817-9b42-ead24207664f" containerName="maas-api" Jul 06 11:55:25.872535 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.872464 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7762871f-1d49-4017-bf75-e811c2a691d0" containerName="istio-proxy" Jul 06 11:55:25.876878 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.876860 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:25.879765 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.879741 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-52f7fb76-openshift-default-dockercfg-hm799\"" Jul 06 11:55:25.887911 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.887850 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz"] Jul 06 11:55:25.984719 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.984688 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff207d8-b8ba-418e-a329-2b2f37d36d04-credential-socket\") pod \"e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:25.985157 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.984723 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff207d8-b8ba-418e-a329-2b2f37d36d04-istio-data\") pod \"e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:25.985157 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.984793 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff207d8-b8ba-418e-a329-2b2f37d36d04-istio-token\") pod \"e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:25.985157 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.984847 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff207d8-b8ba-418e-a329-2b2f37d36d04-workload-socket\") pod \"e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:25.985157 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.984912 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff207d8-b8ba-418e-a329-2b2f37d36d04-istio-envoy\") pod \"e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:25.985157 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.984974 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp985\" (UniqueName: \"kubernetes.io/projected/6ff207d8-b8ba-418e-a329-2b2f37d36d04-kube-api-access-dp985\") pod \"e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:25.985157 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.985007 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff207d8-b8ba-418e-a329-2b2f37d36d04-istiod-ca-cert\") pod \"e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:25.985157 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.985062 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff207d8-b8ba-418e-a329-2b2f37d36d04-workload-certs\") pod \"e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:25.985157 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:25.985086 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff207d8-b8ba-418e-a329-2b2f37d36d04-istio-podinfo\") pod \"e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:26.057071 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:26.057034 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7762871f-1d49-4017-bf75-e811c2a691d0" path="/var/lib/kubelet/pods/7762871f-1d49-4017-bf75-e811c2a691d0/volumes" Jul 06 11:55:26.086415 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:26.086386 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp985\" (UniqueName: \"kubernetes.io/projected/6ff207d8-b8ba-418e-a329-2b2f37d36d04-kube-api-access-dp985\") pod \"e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:26.086540 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:26.086422 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff207d8-b8ba-418e-a329-2b2f37d36d04-istiod-ca-cert\") pod \"e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:26.086540 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:26.086445 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff207d8-b8ba-418e-a329-2b2f37d36d04-workload-certs\") pod \"e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:26.086540 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:26.086459 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff207d8-b8ba-418e-a329-2b2f37d36d04-istio-podinfo\") pod \"e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:26.086655 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:26.086575 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff207d8-b8ba-418e-a329-2b2f37d36d04-credential-socket\") pod \"e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:26.086655 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:26.086602 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff207d8-b8ba-418e-a329-2b2f37d36d04-istio-data\") pod \"e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:26.086655 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:26.086620 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff207d8-b8ba-418e-a329-2b2f37d36d04-istio-token\") pod \"e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:26.086655 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:26.086641 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff207d8-b8ba-418e-a329-2b2f37d36d04-workload-socket\") pod \"e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:26.086823 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:26.086723 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff207d8-b8ba-418e-a329-2b2f37d36d04-istio-envoy\") pod \"e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:26.086940 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:26.086917 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff207d8-b8ba-418e-a329-2b2f37d36d04-workload-certs\") pod \"e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:26.087156 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:26.087135 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff207d8-b8ba-418e-a329-2b2f37d36d04-istio-data\") pod \"e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:26.087232 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:26.087166 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff207d8-b8ba-418e-a329-2b2f37d36d04-credential-socket\") pod \"e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:26.087232 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:26.087200 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff207d8-b8ba-418e-a329-2b2f37d36d04-workload-socket\") pod \"e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:26.087339 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:26.087274 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff207d8-b8ba-418e-a329-2b2f37d36d04-istiod-ca-cert\") pod \"e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:26.088968 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:26.088947 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff207d8-b8ba-418e-a329-2b2f37d36d04-istio-envoy\") pod \"e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:26.089133 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:26.089118 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff207d8-b8ba-418e-a329-2b2f37d36d04-istio-podinfo\") pod \"e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:26.098117 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:26.098097 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp985\" (UniqueName: \"kubernetes.io/projected/6ff207d8-b8ba-418e-a329-2b2f37d36d04-kube-api-access-dp985\") pod \"e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:26.098707 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:26.098666 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff207d8-b8ba-418e-a329-2b2f37d36d04-istio-token\") pod \"e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:26.189838 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:26.189805 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:26.326232 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:26.326205 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz"] Jul 06 11:55:26.327188 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:55:26.327160 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff207d8_b8ba_418e_a329_2b2f37d36d04.slice/crio-57c0dafb3cdd919f3b3a87f30265cbd995859efc494b5b7629a357fdd60f1882 WatchSource:0}: Error finding container 57c0dafb3cdd919f3b3a87f30265cbd995859efc494b5b7629a357fdd60f1882: Status 404 returned error can't find the container with id 57c0dafb3cdd919f3b3a87f30265cbd995859efc494b5b7629a357fdd60f1882 Jul 06 11:55:26.329294 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:26.329259 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 11:55:26.329380 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:26.329328 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 11:55:26.329380 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:26.329354 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 11:55:27.056518 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:27.056481 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" event={"ID":"6ff207d8-b8ba-418e-a329-2b2f37d36d04","Type":"ContainerStarted","Data":"f1bd2d00a8af762d4760dd20eed18f6b8b754b6d8595dc3748682bad6e3294fe"} Jul 06 11:55:27.056894 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:27.056542 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" event={"ID":"6ff207d8-b8ba-418e-a329-2b2f37d36d04","Type":"ContainerStarted","Data":"57c0dafb3cdd919f3b3a87f30265cbd995859efc494b5b7629a357fdd60f1882"} Jul 06 11:55:27.084342 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:27.084273 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" podStartSLOduration=2.084253816 podStartE2EDuration="2.084253816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:55:27.081907149 +0000 UTC m=+1625.561706167" watchObservedRunningTime="2026-07-06 11:55:27.084253816 +0000 UTC m=+1625.564052833" Jul 06 11:55:27.186623 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:27.186578 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-52f7fb76-6769487796-r99td"] Jul 06 11:55:27.190232 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:27.190211 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-52f7fb76-6769487796-r99td" Jul 06 11:55:27.190432 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:27.190403 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:27.193258 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:27.193238 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-52f7fb76-serving-cert\"" Jul 06 11:55:27.195672 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:27.195652 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:27.202904 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:27.202878 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-52f7fb76-6769487796-r99td"] Jul 06 11:55:27.298866 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:27.298809 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cswzs\" (UniqueName: \"kubernetes.io/projected/bf7424f2-35c1-4df7-bb18-a26d3442bacd-kube-api-access-cswzs\") pod \"maas-api-e2e-ait-52f7fb76-6769487796-r99td\" (UID: \"bf7424f2-35c1-4df7-bb18-a26d3442bacd\") " pod="opendatahub/maas-api-e2e-ait-52f7fb76-6769487796-r99td" Jul 06 11:55:27.298866 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:27.298863 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7424f2-35c1-4df7-bb18-a26d3442bacd-openshift-service-ca\") pod \"maas-api-e2e-ait-52f7fb76-6769487796-r99td\" (UID: \"bf7424f2-35c1-4df7-bb18-a26d3442bacd\") " pod="opendatahub/maas-api-e2e-ait-52f7fb76-6769487796-r99td" Jul 06 11:55:27.299079 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:27.299018 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf7424f2-35c1-4df7-bb18-a26d3442bacd-maas-api-tls\") pod \"maas-api-e2e-ait-52f7fb76-6769487796-r99td\" (UID: \"bf7424f2-35c1-4df7-bb18-a26d3442bacd\") " pod="opendatahub/maas-api-e2e-ait-52f7fb76-6769487796-r99td" Jul 06 11:55:27.399827 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:27.399799 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf7424f2-35c1-4df7-bb18-a26d3442bacd-maas-api-tls\") pod \"maas-api-e2e-ait-52f7fb76-6769487796-r99td\" (UID: \"bf7424f2-35c1-4df7-bb18-a26d3442bacd\") " pod="opendatahub/maas-api-e2e-ait-52f7fb76-6769487796-r99td" Jul 06 11:55:27.399992 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:27.399859 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cswzs\" (UniqueName: \"kubernetes.io/projected/bf7424f2-35c1-4df7-bb18-a26d3442bacd-kube-api-access-cswzs\") pod \"maas-api-e2e-ait-52f7fb76-6769487796-r99td\" (UID: \"bf7424f2-35c1-4df7-bb18-a26d3442bacd\") " pod="opendatahub/maas-api-e2e-ait-52f7fb76-6769487796-r99td" Jul 06 11:55:27.399992 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:27.399892 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7424f2-35c1-4df7-bb18-a26d3442bacd-openshift-service-ca\") pod \"maas-api-e2e-ait-52f7fb76-6769487796-r99td\" (UID: \"bf7424f2-35c1-4df7-bb18-a26d3442bacd\") " pod="opendatahub/maas-api-e2e-ait-52f7fb76-6769487796-r99td" Jul 06 11:55:27.399992 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:55:27.399937 2568 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-52f7fb76-serving-cert: secret "maas-api-e2e-ait-52f7fb76-serving-cert" not found Jul 06 11:55:27.400198 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:55:27.399999 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf7424f2-35c1-4df7-bb18-a26d3442bacd-maas-api-tls podName:bf7424f2-35c1-4df7-bb18-a26d3442bacd nodeName:}" failed. No retries permitted until 2026-07-06 11:55:27.899981815 +0000 UTC m=+1626.379780810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bf7424f2-35c1-4df7-bb18-a26d3442bacd-maas-api-tls") pod "maas-api-e2e-ait-52f7fb76-6769487796-r99td" (UID: "bf7424f2-35c1-4df7-bb18-a26d3442bacd") : secret "maas-api-e2e-ait-52f7fb76-serving-cert" not found Jul 06 11:55:27.400577 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:27.400554 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7424f2-35c1-4df7-bb18-a26d3442bacd-openshift-service-ca\") pod \"maas-api-e2e-ait-52f7fb76-6769487796-r99td\" (UID: \"bf7424f2-35c1-4df7-bb18-a26d3442bacd\") " pod="opendatahub/maas-api-e2e-ait-52f7fb76-6769487796-r99td" Jul 06 11:55:27.411700 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:27.411654 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cswzs\" (UniqueName: \"kubernetes.io/projected/bf7424f2-35c1-4df7-bb18-a26d3442bacd-kube-api-access-cswzs\") pod \"maas-api-e2e-ait-52f7fb76-6769487796-r99td\" (UID: \"bf7424f2-35c1-4df7-bb18-a26d3442bacd\") " pod="opendatahub/maas-api-e2e-ait-52f7fb76-6769487796-r99td" Jul 06 11:55:27.905734 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:27.905696 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf7424f2-35c1-4df7-bb18-a26d3442bacd-maas-api-tls\") pod \"maas-api-e2e-ait-52f7fb76-6769487796-r99td\" (UID: \"bf7424f2-35c1-4df7-bb18-a26d3442bacd\") " pod="opendatahub/maas-api-e2e-ait-52f7fb76-6769487796-r99td" Jul 06 11:55:27.908036 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:27.908015 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf7424f2-35c1-4df7-bb18-a26d3442bacd-maas-api-tls\") pod \"maas-api-e2e-ait-52f7fb76-6769487796-r99td\" (UID: \"bf7424f2-35c1-4df7-bb18-a26d3442bacd\") " pod="opendatahub/maas-api-e2e-ait-52f7fb76-6769487796-r99td" Jul 06 11:55:28.060899 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:28.060864 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:28.061861 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:28.061839 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:28.104806 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:28.104774 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-52f7fb76-6769487796-r99td" Jul 06 11:55:28.260959 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:28.260935 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-52f7fb76-6769487796-r99td"] Jul 06 11:55:28.262053 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:55:28.262028 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7424f2_35c1_4df7_bb18_a26d3442bacd.slice/crio-9888e465a4a4ce829ac832f526626cab93c70b417e1164180f5dc3a45d69a18c WatchSource:0}: Error finding container 9888e465a4a4ce829ac832f526626cab93c70b417e1164180f5dc3a45d69a18c: Status 404 returned error can't find the container with id 9888e465a4a4ce829ac832f526626cab93c70b417e1164180f5dc3a45d69a18c Jul 06 11:55:29.067733 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:29.067626 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-52f7fb76-6769487796-r99td" event={"ID":"bf7424f2-35c1-4df7-bb18-a26d3442bacd","Type":"ContainerStarted","Data":"26d6c7fb95c72591d9a588a2456160e2b8d67b4b174ff2a6b35003ff897c77a9"} Jul 06 11:55:29.067733 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:29.067700 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-52f7fb76-6769487796-r99td" event={"ID":"bf7424f2-35c1-4df7-bb18-a26d3442bacd","Type":"ContainerStarted","Data":"9888e465a4a4ce829ac832f526626cab93c70b417e1164180f5dc3a45d69a18c"} Jul 06 11:55:29.067733 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:29.067734 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-52f7fb76-6769487796-r99td" Jul 06 11:55:32.582075 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:32.581185 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-52f7fb76-6769487796-r99td" podStartSLOduration=5.078386855 podStartE2EDuration="5.581161922s" totalImagesPullingTime="502.775067ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:55:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:55:29.086468948 +0000 UTC m=+1627.566267964" watchObservedRunningTime="2026-07-06 11:55:32.581161922 +0000 UTC m=+1631.060960939" Jul 06 11:55:32.583171 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:32.583140 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-52f7fb76-6769487796-r99td"] Jul 06 11:55:32.583407 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:32.583368 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-52f7fb76-6769487796-r99td" podUID="bf7424f2-35c1-4df7-bb18-a26d3442bacd" containerName="maas-api" containerID="cri-o://26d6c7fb95c72591d9a588a2456160e2b8d67b4b174ff2a6b35003ff897c77a9" gracePeriod=30 Jul 06 11:55:32.832993 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:32.832933 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-52f7fb76-6769487796-r99td" Jul 06 11:55:32.949989 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:32.949954 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7424f2-35c1-4df7-bb18-a26d3442bacd-openshift-service-ca\") pod \"bf7424f2-35c1-4df7-bb18-a26d3442bacd\" (UID: \"bf7424f2-35c1-4df7-bb18-a26d3442bacd\") " Jul 06 11:55:32.950167 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:32.950028 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf7424f2-35c1-4df7-bb18-a26d3442bacd-maas-api-tls\") pod \"bf7424f2-35c1-4df7-bb18-a26d3442bacd\" (UID: \"bf7424f2-35c1-4df7-bb18-a26d3442bacd\") " Jul 06 11:55:32.950167 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:32.950071 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cswzs\" (UniqueName: \"kubernetes.io/projected/bf7424f2-35c1-4df7-bb18-a26d3442bacd-kube-api-access-cswzs\") pod \"bf7424f2-35c1-4df7-bb18-a26d3442bacd\" (UID: \"bf7424f2-35c1-4df7-bb18-a26d3442bacd\") " Jul 06 11:55:32.950385 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:32.950360 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf7424f2-35c1-4df7-bb18-a26d3442bacd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bf7424f2-35c1-4df7-bb18-a26d3442bacd" (UID: "bf7424f2-35c1-4df7-bb18-a26d3442bacd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 11:55:32.952170 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:32.952138 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7424f2-35c1-4df7-bb18-a26d3442bacd-kube-api-access-cswzs" (OuterVolumeSpecName: "kube-api-access-cswzs") pod "bf7424f2-35c1-4df7-bb18-a26d3442bacd" (UID: "bf7424f2-35c1-4df7-bb18-a26d3442bacd"). InnerVolumeSpecName "kube-api-access-cswzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 11:55:32.952272 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:32.952220 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf7424f2-35c1-4df7-bb18-a26d3442bacd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bf7424f2-35c1-4df7-bb18-a26d3442bacd" (UID: "bf7424f2-35c1-4df7-bb18-a26d3442bacd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 11:55:33.050945 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:33.050915 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7424f2-35c1-4df7-bb18-a26d3442bacd-openshift-service-ca\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:33.050945 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:33.050941 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf7424f2-35c1-4df7-bb18-a26d3442bacd-maas-api-tls\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:33.050945 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:33.050951 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cswzs\" (UniqueName: \"kubernetes.io/projected/bf7424f2-35c1-4df7-bb18-a26d3442bacd-kube-api-access-cswzs\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:33.089787 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:33.089715 2568 generic.go:356] "Generic (PLEG): container finished" podID="bf7424f2-35c1-4df7-bb18-a26d3442bacd" containerID="26d6c7fb95c72591d9a588a2456160e2b8d67b4b174ff2a6b35003ff897c77a9" exitCode=0 Jul 06 11:55:33.089787 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:33.089775 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-52f7fb76-6769487796-r99td" event={"ID":"bf7424f2-35c1-4df7-bb18-a26d3442bacd","Type":"ContainerDied","Data":"26d6c7fb95c72591d9a588a2456160e2b8d67b4b174ff2a6b35003ff897c77a9"} Jul 06 11:55:33.089938 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:33.089797 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-52f7fb76-6769487796-r99td" Jul 06 11:55:33.089938 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:33.089821 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-52f7fb76-6769487796-r99td" event={"ID":"bf7424f2-35c1-4df7-bb18-a26d3442bacd","Type":"ContainerDied","Data":"9888e465a4a4ce829ac832f526626cab93c70b417e1164180f5dc3a45d69a18c"} Jul 06 11:55:33.089938 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:33.089850 2568 scope.go:117] "RemoveContainer" containerID="26d6c7fb95c72591d9a588a2456160e2b8d67b4b174ff2a6b35003ff897c77a9" Jul 06 11:55:33.099325 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:33.099308 2568 scope.go:117] "RemoveContainer" containerID="26d6c7fb95c72591d9a588a2456160e2b8d67b4b174ff2a6b35003ff897c77a9" Jul 06 11:55:33.099605 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:55:33.099585 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26d6c7fb95c72591d9a588a2456160e2b8d67b4b174ff2a6b35003ff897c77a9\": container with ID starting with 26d6c7fb95c72591d9a588a2456160e2b8d67b4b174ff2a6b35003ff897c77a9 not found: ID does not exist" containerID="26d6c7fb95c72591d9a588a2456160e2b8d67b4b174ff2a6b35003ff897c77a9" Jul 06 11:55:33.099649 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:33.099613 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26d6c7fb95c72591d9a588a2456160e2b8d67b4b174ff2a6b35003ff897c77a9"} err="failed to get container status \"26d6c7fb95c72591d9a588a2456160e2b8d67b4b174ff2a6b35003ff897c77a9\": rpc error: code = NotFound desc = could not find container \"26d6c7fb95c72591d9a588a2456160e2b8d67b4b174ff2a6b35003ff897c77a9\": container with ID starting with 26d6c7fb95c72591d9a588a2456160e2b8d67b4b174ff2a6b35003ff897c77a9 not found: ID does not exist" Jul 06 11:55:33.114080 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:33.114055 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-52f7fb76-6769487796-r99td"] Jul 06 11:55:33.119259 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:33.119238 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-52f7fb76-6769487796-r99td"] Jul 06 11:55:34.052115 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:34.052085 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf7424f2-35c1-4df7-bb18-a26d3442bacd" path="/var/lib/kubelet/pods/bf7424f2-35c1-4df7-bb18-a26d3442bacd/volumes" Jul 06 11:55:38.507603 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:38.507567 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz"] Jul 06 11:55:38.508040 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:38.507786 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" podUID="6ff207d8-b8ba-418e-a329-2b2f37d36d04" containerName="istio-proxy" containerID="cri-o://f1bd2d00a8af762d4760dd20eed18f6b8b754b6d8595dc3748682bad6e3294fe" gracePeriod=30 Jul 06 11:55:43.063770 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:43.063725 2568 patch_prober.go:28] interesting pod/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 11:55:43.064160 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:43.063796 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" podUID="6ff207d8-b8ba-418e-a329-2b2f37d36d04" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 11:55:43.760234 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:43.760212 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:43.840609 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:43.840583 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff207d8-b8ba-418e-a329-2b2f37d36d04-istiod-ca-cert\") pod \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " Jul 06 11:55:43.840818 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:43.840636 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff207d8-b8ba-418e-a329-2b2f37d36d04-istio-podinfo\") pod \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " Jul 06 11:55:43.840818 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:43.840661 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff207d8-b8ba-418e-a329-2b2f37d36d04-istio-token\") pod \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " Jul 06 11:55:43.840818 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:43.840788 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff207d8-b8ba-418e-a329-2b2f37d36d04-istio-envoy\") pod \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " Jul 06 11:55:43.840818 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:43.840811 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff207d8-b8ba-418e-a329-2b2f37d36d04-istio-data\") pod \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " Jul 06 11:55:43.841031 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:43.840835 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff207d8-b8ba-418e-a329-2b2f37d36d04-workload-socket\") pod \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " Jul 06 11:55:43.841031 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:43.840850 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dp985\" (UniqueName: \"kubernetes.io/projected/6ff207d8-b8ba-418e-a329-2b2f37d36d04-kube-api-access-dp985\") pod \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " Jul 06 11:55:43.841149 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:43.841043 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ff207d8-b8ba-418e-a329-2b2f37d36d04-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6ff207d8-b8ba-418e-a329-2b2f37d36d04" (UID: "6ff207d8-b8ba-418e-a329-2b2f37d36d04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 11:55:43.841149 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:43.841088 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff207d8-b8ba-418e-a329-2b2f37d36d04-istio-data" (OuterVolumeSpecName: "istio-data") pod "6ff207d8-b8ba-418e-a329-2b2f37d36d04" (UID: "6ff207d8-b8ba-418e-a329-2b2f37d36d04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:55:43.841149 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:43.841103 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff207d8-b8ba-418e-a329-2b2f37d36d04-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6ff207d8-b8ba-418e-a329-2b2f37d36d04" (UID: "6ff207d8-b8ba-418e-a329-2b2f37d36d04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:55:43.841305 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:43.841201 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff207d8-b8ba-418e-a329-2b2f37d36d04-workload-certs\") pod \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " Jul 06 11:55:43.841305 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:43.841298 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff207d8-b8ba-418e-a329-2b2f37d36d04-credential-socket\") pod \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\" (UID: \"6ff207d8-b8ba-418e-a329-2b2f37d36d04\") " Jul 06 11:55:43.841463 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:43.841441 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff207d8-b8ba-418e-a329-2b2f37d36d04-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6ff207d8-b8ba-418e-a329-2b2f37d36d04" (UID: "6ff207d8-b8ba-418e-a329-2b2f37d36d04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:55:43.841846 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:43.841553 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff207d8-b8ba-418e-a329-2b2f37d36d04-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6ff207d8-b8ba-418e-a329-2b2f37d36d04" (UID: "6ff207d8-b8ba-418e-a329-2b2f37d36d04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:55:43.841846 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:43.841669 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff207d8-b8ba-418e-a329-2b2f37d36d04-credential-socket\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:43.841846 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:43.841721 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ff207d8-b8ba-418e-a329-2b2f37d36d04-istiod-ca-cert\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:43.841846 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:43.841735 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ff207d8-b8ba-418e-a329-2b2f37d36d04-istio-data\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:43.841846 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:43.841749 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ff207d8-b8ba-418e-a329-2b2f37d36d04-workload-socket\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:43.841846 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:43.841763 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ff207d8-b8ba-418e-a329-2b2f37d36d04-workload-certs\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:43.842942 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:43.842910 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ff207d8-b8ba-418e-a329-2b2f37d36d04-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6ff207d8-b8ba-418e-a329-2b2f37d36d04" (UID: "6ff207d8-b8ba-418e-a329-2b2f37d36d04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 11:55:43.843038 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:43.843004 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff207d8-b8ba-418e-a329-2b2f37d36d04-istio-token" (OuterVolumeSpecName: "istio-token") pod "6ff207d8-b8ba-418e-a329-2b2f37d36d04" (UID: "6ff207d8-b8ba-418e-a329-2b2f37d36d04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 11:55:43.843083 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:43.843031 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ff207d8-b8ba-418e-a329-2b2f37d36d04-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6ff207d8-b8ba-418e-a329-2b2f37d36d04" (UID: "6ff207d8-b8ba-418e-a329-2b2f37d36d04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:55:43.843483 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:43.843460 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff207d8-b8ba-418e-a329-2b2f37d36d04-kube-api-access-dp985" (OuterVolumeSpecName: "kube-api-access-dp985") pod "6ff207d8-b8ba-418e-a329-2b2f37d36d04" (UID: "6ff207d8-b8ba-418e-a329-2b2f37d36d04"). InnerVolumeSpecName "kube-api-access-dp985". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 11:55:43.942852 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:43.942818 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ff207d8-b8ba-418e-a329-2b2f37d36d04-istio-podinfo\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:43.942852 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:43.942847 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ff207d8-b8ba-418e-a329-2b2f37d36d04-istio-token\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:43.942852 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:43.942856 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ff207d8-b8ba-418e-a329-2b2f37d36d04-istio-envoy\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:43.943100 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:43.942864 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp985\" (UniqueName: \"kubernetes.io/projected/6ff207d8-b8ba-418e-a329-2b2f37d36d04-kube-api-access-dp985\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:44.149154 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.149116 2568 generic.go:356] "Generic (PLEG): container finished" podID="6ff207d8-b8ba-418e-a329-2b2f37d36d04" containerID="f1bd2d00a8af762d4760dd20eed18f6b8b754b6d8595dc3748682bad6e3294fe" exitCode=0 Jul 06 11:55:44.149636 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.149188 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" event={"ID":"6ff207d8-b8ba-418e-a329-2b2f37d36d04","Type":"ContainerDied","Data":"f1bd2d00a8af762d4760dd20eed18f6b8b754b6d8595dc3748682bad6e3294fe"} Jul 06 11:55:44.149636 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.149191 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" Jul 06 11:55:44.149636 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.149235 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz" event={"ID":"6ff207d8-b8ba-418e-a329-2b2f37d36d04","Type":"ContainerDied","Data":"57c0dafb3cdd919f3b3a87f30265cbd995859efc494b5b7629a357fdd60f1882"} Jul 06 11:55:44.149636 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.149258 2568 scope.go:117] "RemoveContainer" containerID="f1bd2d00a8af762d4760dd20eed18f6b8b754b6d8595dc3748682bad6e3294fe" Jul 06 11:55:44.158249 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.158227 2568 scope.go:117] "RemoveContainer" containerID="f1bd2d00a8af762d4760dd20eed18f6b8b754b6d8595dc3748682bad6e3294fe" Jul 06 11:55:44.158497 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:55:44.158476 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1bd2d00a8af762d4760dd20eed18f6b8b754b6d8595dc3748682bad6e3294fe\": container with ID starting with f1bd2d00a8af762d4760dd20eed18f6b8b754b6d8595dc3748682bad6e3294fe not found: ID does not exist" containerID="f1bd2d00a8af762d4760dd20eed18f6b8b754b6d8595dc3748682bad6e3294fe" Jul 06 11:55:44.158566 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.158505 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1bd2d00a8af762d4760dd20eed18f6b8b754b6d8595dc3748682bad6e3294fe"} err="failed to get container status \"f1bd2d00a8af762d4760dd20eed18f6b8b754b6d8595dc3748682bad6e3294fe\": rpc error: code = NotFound desc = could not find container \"f1bd2d00a8af762d4760dd20eed18f6b8b754b6d8595dc3748682bad6e3294fe\": container with ID starting with f1bd2d00a8af762d4760dd20eed18f6b8b754b6d8595dc3748682bad6e3294fe not found: ID does not exist" Jul 06 11:55:44.169529 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.169507 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz"] Jul 06 11:55:44.174666 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.174646 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-52f7fb76-openshift-default-7c8c6ccccc-8xdtz"] Jul 06 11:55:44.793210 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.793179 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq"] Jul 06 11:55:44.793546 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.793529 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf7424f2-35c1-4df7-bb18-a26d3442bacd" containerName="maas-api" Jul 06 11:55:44.793600 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.793551 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf7424f2-35c1-4df7-bb18-a26d3442bacd" containerName="maas-api" Jul 06 11:55:44.793600 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.793597 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ff207d8-b8ba-418e-a329-2b2f37d36d04" containerName="istio-proxy" Jul 06 11:55:44.793692 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.793607 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ff207d8-b8ba-418e-a329-2b2f37d36d04" containerName="istio-proxy" Jul 06 11:55:44.793734 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.793715 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf7424f2-35c1-4df7-bb18-a26d3442bacd" containerName="maas-api" Jul 06 11:55:44.793734 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.793728 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ff207d8-b8ba-418e-a329-2b2f37d36d04" containerName="istio-proxy" Jul 06 11:55:44.798032 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.798000 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:44.800621 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.800589 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-aee6826c-openshift-default-dockercfg-dr85x\"" Jul 06 11:55:44.807900 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.807878 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq"] Jul 06 11:55:44.851582 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.851552 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a172a51-7c38-4b38-8363-a7a56e5660bb-credential-socket\") pod \"e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:44.851753 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.851600 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a172a51-7c38-4b38-8363-a7a56e5660bb-workload-certs\") pod \"e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:44.851753 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.851620 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a172a51-7c38-4b38-8363-a7a56e5660bb-istiod-ca-cert\") pod \"e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:44.851753 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.851704 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccl5x\" (UniqueName: \"kubernetes.io/projected/1a172a51-7c38-4b38-8363-a7a56e5660bb-kube-api-access-ccl5x\") pod \"e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:44.851899 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.851753 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a172a51-7c38-4b38-8363-a7a56e5660bb-workload-socket\") pod \"e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:44.851899 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.851775 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a172a51-7c38-4b38-8363-a7a56e5660bb-istio-data\") pod \"e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:44.851899 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.851790 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a172a51-7c38-4b38-8363-a7a56e5660bb-istio-podinfo\") pod \"e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:44.851899 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.851821 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a172a51-7c38-4b38-8363-a7a56e5660bb-istio-token\") pod \"e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:44.851899 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.851890 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a172a51-7c38-4b38-8363-a7a56e5660bb-istio-envoy\") pod \"e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:44.952924 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.952891 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a172a51-7c38-4b38-8363-a7a56e5660bb-credential-socket\") pod \"e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:44.953096 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.952939 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a172a51-7c38-4b38-8363-a7a56e5660bb-workload-certs\") pod \"e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:44.953096 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.952965 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a172a51-7c38-4b38-8363-a7a56e5660bb-istiod-ca-cert\") pod \"e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:44.953096 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.953002 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccl5x\" (UniqueName: \"kubernetes.io/projected/1a172a51-7c38-4b38-8363-a7a56e5660bb-kube-api-access-ccl5x\") pod \"e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:44.953096 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.953030 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a172a51-7c38-4b38-8363-a7a56e5660bb-workload-socket\") pod \"e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:44.953096 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.953059 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a172a51-7c38-4b38-8363-a7a56e5660bb-istio-data\") pod \"e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:44.953096 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.953082 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a172a51-7c38-4b38-8363-a7a56e5660bb-istio-podinfo\") pod \"e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:44.953465 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.953114 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a172a51-7c38-4b38-8363-a7a56e5660bb-istio-token\") pod \"e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:44.953465 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.953293 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a172a51-7c38-4b38-8363-a7a56e5660bb-istio-envoy\") pod \"e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:44.953465 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.953385 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a172a51-7c38-4b38-8363-a7a56e5660bb-workload-certs\") pod \"e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:44.953465 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.953429 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a172a51-7c38-4b38-8363-a7a56e5660bb-istio-data\") pod \"e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:44.953720 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.953667 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a172a51-7c38-4b38-8363-a7a56e5660bb-credential-socket\") pod \"e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:44.953809 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.953789 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a172a51-7c38-4b38-8363-a7a56e5660bb-workload-socket\") pod \"e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:44.953849 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.953789 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a172a51-7c38-4b38-8363-a7a56e5660bb-istiod-ca-cert\") pod \"e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:44.955464 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.955434 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a172a51-7c38-4b38-8363-a7a56e5660bb-istio-envoy\") pod \"e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:44.955624 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.955607 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a172a51-7c38-4b38-8363-a7a56e5660bb-istio-podinfo\") pod \"e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:44.961520 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.961498 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a172a51-7c38-4b38-8363-a7a56e5660bb-istio-token\") pod \"e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:44.961669 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:44.961650 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccl5x\" (UniqueName: \"kubernetes.io/projected/1a172a51-7c38-4b38-8363-a7a56e5660bb-kube-api-access-ccl5x\") pod \"e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:45.110151 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:45.110065 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:45.247187 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:45.247159 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq"] Jul 06 11:55:45.248644 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:55:45.248611 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a172a51_7c38_4b38_8363_a7a56e5660bb.slice/crio-b69662bd7afba7bac9a3a1aa04a409aeb2b9f0d64b98f0759e7691e63e2f73d4 WatchSource:0}: Error finding container b69662bd7afba7bac9a3a1aa04a409aeb2b9f0d64b98f0759e7691e63e2f73d4: Status 404 returned error can't find the container with id b69662bd7afba7bac9a3a1aa04a409aeb2b9f0d64b98f0759e7691e63e2f73d4 Jul 06 11:55:45.250575 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:45.250542 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 11:55:45.250658 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:45.250606 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 11:55:45.250658 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:45.250635 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 11:55:46.052396 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:46.052365 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ff207d8-b8ba-418e-a329-2b2f37d36d04" path="/var/lib/kubelet/pods/6ff207d8-b8ba-418e-a329-2b2f37d36d04/volumes" Jul 06 11:55:46.101718 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:46.101685 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b"] Jul 06 11:55:46.105228 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:46.105207 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b" Jul 06 11:55:46.108260 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:46.108240 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-aee6826c-serving-cert\"" Jul 06 11:55:46.116466 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:46.116445 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b"] Jul 06 11:55:46.161138 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:46.161106 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" event={"ID":"1a172a51-7c38-4b38-8363-a7a56e5660bb","Type":"ContainerStarted","Data":"4e4b3874a138d18e4e3f2cbac038324a1b9d36e6bc5513dab0cb27cab6fff1bf"} Jul 06 11:55:46.161303 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:46.161155 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" event={"ID":"1a172a51-7c38-4b38-8363-a7a56e5660bb","Type":"ContainerStarted","Data":"b69662bd7afba7bac9a3a1aa04a409aeb2b9f0d64b98f0759e7691e63e2f73d4"} Jul 06 11:55:46.164669 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:46.164648 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49cd0d05-094c-44ad-85f7-14f43734cf53-openshift-service-ca\") pod \"maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b\" (UID: \"49cd0d05-094c-44ad-85f7-14f43734cf53\") " pod="opendatahub/maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b" Jul 06 11:55:46.164758 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:46.164717 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49cd0d05-094c-44ad-85f7-14f43734cf53-maas-api-tls\") pod \"maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b\" (UID: \"49cd0d05-094c-44ad-85f7-14f43734cf53\") " pod="opendatahub/maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b" Jul 06 11:55:46.164808 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:46.164781 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxvg4\" (UniqueName: \"kubernetes.io/projected/49cd0d05-094c-44ad-85f7-14f43734cf53-kube-api-access-sxvg4\") pod \"maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b\" (UID: \"49cd0d05-094c-44ad-85f7-14f43734cf53\") " pod="opendatahub/maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b" Jul 06 11:55:46.184866 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:46.184821 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" podStartSLOduration=2.184804498 podStartE2EDuration="2.184804498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:55:46.183275732 +0000 UTC m=+1644.663074748" watchObservedRunningTime="2026-07-06 11:55:46.184804498 +0000 UTC m=+1644.664603520" Jul 06 11:55:46.265776 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:46.265738 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxvg4\" (UniqueName: \"kubernetes.io/projected/49cd0d05-094c-44ad-85f7-14f43734cf53-kube-api-access-sxvg4\") pod \"maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b\" (UID: \"49cd0d05-094c-44ad-85f7-14f43734cf53\") " pod="opendatahub/maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b" Jul 06 11:55:46.266259 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:46.265807 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49cd0d05-094c-44ad-85f7-14f43734cf53-openshift-service-ca\") pod \"maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b\" (UID: \"49cd0d05-094c-44ad-85f7-14f43734cf53\") " pod="opendatahub/maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b" Jul 06 11:55:46.266259 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:46.265857 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49cd0d05-094c-44ad-85f7-14f43734cf53-maas-api-tls\") pod \"maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b\" (UID: \"49cd0d05-094c-44ad-85f7-14f43734cf53\") " pod="opendatahub/maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b" Jul 06 11:55:46.266259 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:55:46.265962 2568 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-aee6826c-serving-cert: secret "maas-api-e2e-derive-aee6826c-serving-cert" not found Jul 06 11:55:46.266259 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:55:46.266072 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49cd0d05-094c-44ad-85f7-14f43734cf53-maas-api-tls podName:49cd0d05-094c-44ad-85f7-14f43734cf53 nodeName:}" failed. No retries permitted until 2026-07-06 11:55:46.766000596 +0000 UTC m=+1645.245799589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/49cd0d05-094c-44ad-85f7-14f43734cf53-maas-api-tls") pod "maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b" (UID: "49cd0d05-094c-44ad-85f7-14f43734cf53") : secret "maas-api-e2e-derive-aee6826c-serving-cert" not found Jul 06 11:55:46.266445 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:46.266424 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49cd0d05-094c-44ad-85f7-14f43734cf53-openshift-service-ca\") pod \"maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b\" (UID: \"49cd0d05-094c-44ad-85f7-14f43734cf53\") " pod="opendatahub/maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b" Jul 06 11:55:46.276610 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:46.276587 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxvg4\" (UniqueName: \"kubernetes.io/projected/49cd0d05-094c-44ad-85f7-14f43734cf53-kube-api-access-sxvg4\") pod \"maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b\" (UID: \"49cd0d05-094c-44ad-85f7-14f43734cf53\") " pod="opendatahub/maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b" Jul 06 11:55:46.778069 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:46.778030 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49cd0d05-094c-44ad-85f7-14f43734cf53-maas-api-tls\") pod \"maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b\" (UID: \"49cd0d05-094c-44ad-85f7-14f43734cf53\") " pod="opendatahub/maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b" Jul 06 11:55:46.780433 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:46.780404 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49cd0d05-094c-44ad-85f7-14f43734cf53-maas-api-tls\") pod \"maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b\" (UID: \"49cd0d05-094c-44ad-85f7-14f43734cf53\") " pod="opendatahub/maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b" Jul 06 11:55:47.017546 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:47.017502 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b" Jul 06 11:55:47.111420 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:47.111375 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:47.116208 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:47.116180 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:47.162261 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:47.162237 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b"] Jul 06 11:55:47.164261 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:55:47.164234 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49cd0d05_094c_44ad_85f7_14f43734cf53.slice/crio-6c0ab3008141a9e790dcccc0bb5da9c3c1a4a06962054de4fd3b8623b513a326 WatchSource:0}: Error finding container 6c0ab3008141a9e790dcccc0bb5da9c3c1a4a06962054de4fd3b8623b513a326: Status 404 returned error can't find the container with id 6c0ab3008141a9e790dcccc0bb5da9c3c1a4a06962054de4fd3b8623b513a326 Jul 06 11:55:47.168188 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:47.168167 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:47.169174 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:47.169103 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:48.175145 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:48.175110 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b" event={"ID":"49cd0d05-094c-44ad-85f7-14f43734cf53","Type":"ContainerStarted","Data":"eb9cd8281c8fed4a38e39ae6f12eb14cd218ac6a7629c05603313d3a477e92e4"} Jul 06 11:55:48.175554 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:48.175176 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b" event={"ID":"49cd0d05-094c-44ad-85f7-14f43734cf53","Type":"ContainerStarted","Data":"6c0ab3008141a9e790dcccc0bb5da9c3c1a4a06962054de4fd3b8623b513a326"} Jul 06 11:55:48.175554 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:48.175456 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b" Jul 06 11:55:48.195133 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:48.195088 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b" podStartSLOduration=1.885408864 podStartE2EDuration="2.195074536s" totalImagesPullingTime="309.665672ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:55:48.192889653 +0000 UTC m=+1646.672688669" watchObservedRunningTime="2026-07-06 11:55:48.195074536 +0000 UTC m=+1646.674873561" Jul 06 11:55:50.612305 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:50.612273 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq"] Jul 06 11:55:50.612833 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:50.612529 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" podUID="1a172a51-7c38-4b38-8363-a7a56e5660bb" containerName="istio-proxy" containerID="cri-o://4e4b3874a138d18e4e3f2cbac038324a1b9d36e6bc5513dab0cb27cab6fff1bf" gracePeriod=30 Jul 06 11:55:50.835770 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:50.835731 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b"] Jul 06 11:55:50.836287 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:50.836256 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b" podUID="49cd0d05-094c-44ad-85f7-14f43734cf53" containerName="maas-api" containerID="cri-o://eb9cd8281c8fed4a38e39ae6f12eb14cd218ac6a7629c05603313d3a477e92e4" gracePeriod=30 Jul 06 11:55:51.182256 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:51.182234 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b" Jul 06 11:55:51.193647 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:51.193617 2568 generic.go:356] "Generic (PLEG): container finished" podID="49cd0d05-094c-44ad-85f7-14f43734cf53" containerID="eb9cd8281c8fed4a38e39ae6f12eb14cd218ac6a7629c05603313d3a477e92e4" exitCode=0 Jul 06 11:55:51.193821 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:51.193766 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b" event={"ID":"49cd0d05-094c-44ad-85f7-14f43734cf53","Type":"ContainerDied","Data":"eb9cd8281c8fed4a38e39ae6f12eb14cd218ac6a7629c05603313d3a477e92e4"} Jul 06 11:55:51.193887 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:51.193823 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b" event={"ID":"49cd0d05-094c-44ad-85f7-14f43734cf53","Type":"ContainerDied","Data":"6c0ab3008141a9e790dcccc0bb5da9c3c1a4a06962054de4fd3b8623b513a326"} Jul 06 11:55:51.193887 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:51.193828 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b" Jul 06 11:55:51.193887 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:51.193853 2568 scope.go:117] "RemoveContainer" containerID="eb9cd8281c8fed4a38e39ae6f12eb14cd218ac6a7629c05603313d3a477e92e4" Jul 06 11:55:51.204361 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:51.204321 2568 scope.go:117] "RemoveContainer" containerID="eb9cd8281c8fed4a38e39ae6f12eb14cd218ac6a7629c05603313d3a477e92e4" Jul 06 11:55:51.204689 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:55:51.204644 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb9cd8281c8fed4a38e39ae6f12eb14cd218ac6a7629c05603313d3a477e92e4\": container with ID starting with eb9cd8281c8fed4a38e39ae6f12eb14cd218ac6a7629c05603313d3a477e92e4 not found: ID does not exist" containerID="eb9cd8281c8fed4a38e39ae6f12eb14cd218ac6a7629c05603313d3a477e92e4" Jul 06 11:55:51.204801 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:51.204697 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb9cd8281c8fed4a38e39ae6f12eb14cd218ac6a7629c05603313d3a477e92e4"} err="failed to get container status \"eb9cd8281c8fed4a38e39ae6f12eb14cd218ac6a7629c05603313d3a477e92e4\": rpc error: code = NotFound desc = could not find container \"eb9cd8281c8fed4a38e39ae6f12eb14cd218ac6a7629c05603313d3a477e92e4\": container with ID starting with eb9cd8281c8fed4a38e39ae6f12eb14cd218ac6a7629c05603313d3a477e92e4 not found: ID does not exist" Jul 06 11:55:51.224390 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:51.224368 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxvg4\" (UniqueName: \"kubernetes.io/projected/49cd0d05-094c-44ad-85f7-14f43734cf53-kube-api-access-sxvg4\") pod \"49cd0d05-094c-44ad-85f7-14f43734cf53\" (UID: \"49cd0d05-094c-44ad-85f7-14f43734cf53\") " Jul 06 11:55:51.224511 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:51.224439 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49cd0d05-094c-44ad-85f7-14f43734cf53-openshift-service-ca\") pod \"49cd0d05-094c-44ad-85f7-14f43734cf53\" (UID: \"49cd0d05-094c-44ad-85f7-14f43734cf53\") " Jul 06 11:55:51.224511 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:51.224462 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49cd0d05-094c-44ad-85f7-14f43734cf53-maas-api-tls\") pod \"49cd0d05-094c-44ad-85f7-14f43734cf53\" (UID: \"49cd0d05-094c-44ad-85f7-14f43734cf53\") " Jul 06 11:55:51.224842 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:51.224818 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49cd0d05-094c-44ad-85f7-14f43734cf53-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "49cd0d05-094c-44ad-85f7-14f43734cf53" (UID: "49cd0d05-094c-44ad-85f7-14f43734cf53"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 11:55:51.226506 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:51.226471 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49cd0d05-094c-44ad-85f7-14f43734cf53-kube-api-access-sxvg4" (OuterVolumeSpecName: "kube-api-access-sxvg4") pod "49cd0d05-094c-44ad-85f7-14f43734cf53" (UID: "49cd0d05-094c-44ad-85f7-14f43734cf53"). InnerVolumeSpecName "kube-api-access-sxvg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 11:55:51.226581 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:51.226481 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49cd0d05-094c-44ad-85f7-14f43734cf53-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "49cd0d05-094c-44ad-85f7-14f43734cf53" (UID: "49cd0d05-094c-44ad-85f7-14f43734cf53"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 11:55:51.325400 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:51.325372 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxvg4\" (UniqueName: \"kubernetes.io/projected/49cd0d05-094c-44ad-85f7-14f43734cf53-kube-api-access-sxvg4\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:51.325400 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:51.325396 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49cd0d05-094c-44ad-85f7-14f43734cf53-openshift-service-ca\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:51.325400 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:51.325405 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49cd0d05-094c-44ad-85f7-14f43734cf53-maas-api-tls\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:51.518528 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:51.518499 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b"] Jul 06 11:55:51.522035 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:51.522010 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-aee6826c-57c74b6b5d-79g2b"] Jul 06 11:55:52.057001 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:52.056964 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49cd0d05-094c-44ad-85f7-14f43734cf53" path="/var/lib/kubelet/pods/49cd0d05-094c-44ad-85f7-14f43734cf53/volumes" Jul 06 11:55:56.897276 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:56.897248 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:56.980364 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:56.980285 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a172a51-7c38-4b38-8363-a7a56e5660bb-credential-socket\") pod \"1a172a51-7c38-4b38-8363-a7a56e5660bb\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " Jul 06 11:55:56.980364 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:56.980319 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a172a51-7c38-4b38-8363-a7a56e5660bb-workload-certs\") pod \"1a172a51-7c38-4b38-8363-a7a56e5660bb\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " Jul 06 11:55:56.980364 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:56.980336 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a172a51-7c38-4b38-8363-a7a56e5660bb-istiod-ca-cert\") pod \"1a172a51-7c38-4b38-8363-a7a56e5660bb\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " Jul 06 11:55:56.980364 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:56.980353 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a172a51-7c38-4b38-8363-a7a56e5660bb-workload-socket\") pod \"1a172a51-7c38-4b38-8363-a7a56e5660bb\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " Jul 06 11:55:56.980719 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:56.980372 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a172a51-7c38-4b38-8363-a7a56e5660bb-istio-token\") pod \"1a172a51-7c38-4b38-8363-a7a56e5660bb\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " Jul 06 11:55:56.980719 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:56.980420 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a172a51-7c38-4b38-8363-a7a56e5660bb-istio-data\") pod \"1a172a51-7c38-4b38-8363-a7a56e5660bb\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " Jul 06 11:55:56.980719 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:56.980447 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a172a51-7c38-4b38-8363-a7a56e5660bb-istio-envoy\") pod \"1a172a51-7c38-4b38-8363-a7a56e5660bb\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " Jul 06 11:55:56.980719 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:56.980541 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a172a51-7c38-4b38-8363-a7a56e5660bb-istio-podinfo\") pod \"1a172a51-7c38-4b38-8363-a7a56e5660bb\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " Jul 06 11:55:56.980719 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:56.980570 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccl5x\" (UniqueName: \"kubernetes.io/projected/1a172a51-7c38-4b38-8363-a7a56e5660bb-kube-api-access-ccl5x\") pod \"1a172a51-7c38-4b38-8363-a7a56e5660bb\" (UID: \"1a172a51-7c38-4b38-8363-a7a56e5660bb\") " Jul 06 11:55:56.980719 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:56.980584 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a172a51-7c38-4b38-8363-a7a56e5660bb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1a172a51-7c38-4b38-8363-a7a56e5660bb" (UID: "1a172a51-7c38-4b38-8363-a7a56e5660bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:55:56.980719 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:56.980617 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a172a51-7c38-4b38-8363-a7a56e5660bb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1a172a51-7c38-4b38-8363-a7a56e5660bb" (UID: "1a172a51-7c38-4b38-8363-a7a56e5660bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:55:56.980719 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:56.980631 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a172a51-7c38-4b38-8363-a7a56e5660bb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1a172a51-7c38-4b38-8363-a7a56e5660bb" (UID: "1a172a51-7c38-4b38-8363-a7a56e5660bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:55:56.981077 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:56.980731 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a172a51-7c38-4b38-8363-a7a56e5660bb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1a172a51-7c38-4b38-8363-a7a56e5660bb" (UID: "1a172a51-7c38-4b38-8363-a7a56e5660bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 11:55:56.981077 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:56.980895 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a172a51-7c38-4b38-8363-a7a56e5660bb-credential-socket\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:56.981077 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:56.980915 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a172a51-7c38-4b38-8363-a7a56e5660bb-workload-certs\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:56.981077 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:56.980928 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a172a51-7c38-4b38-8363-a7a56e5660bb-istiod-ca-cert\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:56.981077 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:56.980940 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a172a51-7c38-4b38-8363-a7a56e5660bb-workload-socket\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:56.981285 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:56.981244 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a172a51-7c38-4b38-8363-a7a56e5660bb-istio-data" (OuterVolumeSpecName: "istio-data") pod "1a172a51-7c38-4b38-8363-a7a56e5660bb" (UID: "1a172a51-7c38-4b38-8363-a7a56e5660bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:55:56.983144 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:56.983108 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a172a51-7c38-4b38-8363-a7a56e5660bb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1a172a51-7c38-4b38-8363-a7a56e5660bb" (UID: "1a172a51-7c38-4b38-8363-a7a56e5660bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:55:56.983144 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:56.983110 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a172a51-7c38-4b38-8363-a7a56e5660bb-istio-token" (OuterVolumeSpecName: "istio-token") pod "1a172a51-7c38-4b38-8363-a7a56e5660bb" (UID: "1a172a51-7c38-4b38-8363-a7a56e5660bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 11:55:56.983299 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:56.983173 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a172a51-7c38-4b38-8363-a7a56e5660bb-kube-api-access-ccl5x" (OuterVolumeSpecName: "kube-api-access-ccl5x") pod "1a172a51-7c38-4b38-8363-a7a56e5660bb" (UID: "1a172a51-7c38-4b38-8363-a7a56e5660bb"). InnerVolumeSpecName "kube-api-access-ccl5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 11:55:56.983299 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:56.983177 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a172a51-7c38-4b38-8363-a7a56e5660bb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1a172a51-7c38-4b38-8363-a7a56e5660bb" (UID: "1a172a51-7c38-4b38-8363-a7a56e5660bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 11:55:57.081716 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:57.081666 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a172a51-7c38-4b38-8363-a7a56e5660bb-istio-data\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:57.081716 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:57.081710 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a172a51-7c38-4b38-8363-a7a56e5660bb-istio-envoy\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:57.081716 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:57.081719 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a172a51-7c38-4b38-8363-a7a56e5660bb-istio-podinfo\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:57.081941 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:57.081727 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccl5x\" (UniqueName: \"kubernetes.io/projected/1a172a51-7c38-4b38-8363-a7a56e5660bb-kube-api-access-ccl5x\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:57.081941 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:57.081737 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a172a51-7c38-4b38-8363-a7a56e5660bb-istio-token\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:55:57.231033 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:57.230911 2568 generic.go:356] "Generic (PLEG): container finished" podID="1a172a51-7c38-4b38-8363-a7a56e5660bb" containerID="4e4b3874a138d18e4e3f2cbac038324a1b9d36e6bc5513dab0cb27cab6fff1bf" exitCode=0 Jul 06 11:55:57.231033 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:57.230952 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" event={"ID":"1a172a51-7c38-4b38-8363-a7a56e5660bb","Type":"ContainerDied","Data":"4e4b3874a138d18e4e3f2cbac038324a1b9d36e6bc5513dab0cb27cab6fff1bf"} Jul 06 11:55:57.231033 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:57.230968 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" Jul 06 11:55:57.231033 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:57.230998 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq" event={"ID":"1a172a51-7c38-4b38-8363-a7a56e5660bb","Type":"ContainerDied","Data":"b69662bd7afba7bac9a3a1aa04a409aeb2b9f0d64b98f0759e7691e63e2f73d4"} Jul 06 11:55:57.231033 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:57.231019 2568 scope.go:117] "RemoveContainer" containerID="4e4b3874a138d18e4e3f2cbac038324a1b9d36e6bc5513dab0cb27cab6fff1bf" Jul 06 11:55:57.240556 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:57.240535 2568 scope.go:117] "RemoveContainer" containerID="4e4b3874a138d18e4e3f2cbac038324a1b9d36e6bc5513dab0cb27cab6fff1bf" Jul 06 11:55:57.240849 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:55:57.240819 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e4b3874a138d18e4e3f2cbac038324a1b9d36e6bc5513dab0cb27cab6fff1bf\": container with ID starting with 4e4b3874a138d18e4e3f2cbac038324a1b9d36e6bc5513dab0cb27cab6fff1bf not found: ID does not exist" containerID="4e4b3874a138d18e4e3f2cbac038324a1b9d36e6bc5513dab0cb27cab6fff1bf" Jul 06 11:55:57.240925 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:57.240853 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e4b3874a138d18e4e3f2cbac038324a1b9d36e6bc5513dab0cb27cab6fff1bf"} err="failed to get container status \"4e4b3874a138d18e4e3f2cbac038324a1b9d36e6bc5513dab0cb27cab6fff1bf\": rpc error: code = NotFound desc = could not find container \"4e4b3874a138d18e4e3f2cbac038324a1b9d36e6bc5513dab0cb27cab6fff1bf\": container with ID starting with 4e4b3874a138d18e4e3f2cbac038324a1b9d36e6bc5513dab0cb27cab6fff1bf not found: ID does not exist" Jul 06 11:55:57.298417 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:57.298390 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq"] Jul 06 11:55:57.310524 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:57.310497 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-aee6826c-openshift-default-76cb44bccf-6xxtq"] Jul 06 11:55:58.052618 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:55:58.052583 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a172a51-7c38-4b38-8363-a7a56e5660bb" path="/var/lib/kubelet/pods/1a172a51-7c38-4b38-8363-a7a56e5660bb/volumes" Jul 06 11:57:32.647958 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.647919 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf"] Jul 06 11:57:32.648443 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.648269 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49cd0d05-094c-44ad-85f7-14f43734cf53" containerName="maas-api" Jul 06 11:57:32.648443 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.648282 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="49cd0d05-094c-44ad-85f7-14f43734cf53" containerName="maas-api" Jul 06 11:57:32.648443 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.648304 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a172a51-7c38-4b38-8363-a7a56e5660bb" containerName="istio-proxy" Jul 06 11:57:32.648443 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.648310 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a172a51-7c38-4b38-8363-a7a56e5660bb" containerName="istio-proxy" Jul 06 11:57:32.648443 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.648364 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a172a51-7c38-4b38-8363-a7a56e5660bb" containerName="istio-proxy" Jul 06 11:57:32.648443 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.648373 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="49cd0d05-094c-44ad-85f7-14f43734cf53" containerName="maas-api" Jul 06 11:57:32.651321 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.651305 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:32.653907 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.653884 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-07d1fc4e-openshift-default-dockercfg-7gh8c\"" Jul 06 11:57:32.673945 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.673912 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf"] Jul 06 11:57:32.681810 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.681785 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e57a8569-877d-48ac-82e4-6c5d2ace9f99-workload-certs\") pod \"e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:32.681936 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.681817 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e57a8569-877d-48ac-82e4-6c5d2ace9f99-istiod-ca-cert\") pod \"e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:32.681936 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.681837 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6vpf\" (UniqueName: \"kubernetes.io/projected/e57a8569-877d-48ac-82e4-6c5d2ace9f99-kube-api-access-g6vpf\") pod \"e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:32.681936 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.681858 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e57a8569-877d-48ac-82e4-6c5d2ace9f99-istio-podinfo\") pod \"e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:32.682110 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.681939 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57a8569-877d-48ac-82e4-6c5d2ace9f99-credential-socket\") pod \"e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:32.682110 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.681982 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e57a8569-877d-48ac-82e4-6c5d2ace9f99-istio-token\") pod \"e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:32.682110 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.682035 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57a8569-877d-48ac-82e4-6c5d2ace9f99-workload-socket\") pod \"e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:32.682110 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.682058 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e57a8569-877d-48ac-82e4-6c5d2ace9f99-istio-data\") pod \"e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:32.682310 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.682125 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e57a8569-877d-48ac-82e4-6c5d2ace9f99-istio-envoy\") pod \"e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:32.782873 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.782837 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e57a8569-877d-48ac-82e4-6c5d2ace9f99-workload-certs\") pod \"e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:32.782873 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.782875 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e57a8569-877d-48ac-82e4-6c5d2ace9f99-istiod-ca-cert\") pod \"e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:32.783101 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.782895 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6vpf\" (UniqueName: \"kubernetes.io/projected/e57a8569-877d-48ac-82e4-6c5d2ace9f99-kube-api-access-g6vpf\") pod \"e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:32.783101 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.782914 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e57a8569-877d-48ac-82e4-6c5d2ace9f99-istio-podinfo\") pod \"e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:32.783101 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.782937 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57a8569-877d-48ac-82e4-6c5d2ace9f99-credential-socket\") pod \"e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:32.783101 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.782964 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e57a8569-877d-48ac-82e4-6c5d2ace9f99-istio-token\") pod \"e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:32.783101 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.782999 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57a8569-877d-48ac-82e4-6c5d2ace9f99-workload-socket\") pod \"e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:32.783101 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.783027 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e57a8569-877d-48ac-82e4-6c5d2ace9f99-istio-data\") pod \"e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:32.783101 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.783088 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e57a8569-877d-48ac-82e4-6c5d2ace9f99-istio-envoy\") pod \"e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:32.783469 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.783390 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57a8569-877d-48ac-82e4-6c5d2ace9f99-credential-socket\") pod \"e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:32.783517 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.783466 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e57a8569-877d-48ac-82e4-6c5d2ace9f99-workload-certs\") pod \"e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:32.783517 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.783504 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e57a8569-877d-48ac-82e4-6c5d2ace9f99-istio-data\") pod \"e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:32.783630 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.783596 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57a8569-877d-48ac-82e4-6c5d2ace9f99-workload-socket\") pod \"e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:32.783746 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.783640 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e57a8569-877d-48ac-82e4-6c5d2ace9f99-istiod-ca-cert\") pod \"e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:32.785400 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.785374 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e57a8569-877d-48ac-82e4-6c5d2ace9f99-istio-envoy\") pod \"e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:32.785561 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.785544 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e57a8569-877d-48ac-82e4-6c5d2ace9f99-istio-podinfo\") pod \"e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:32.791767 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.791740 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6vpf\" (UniqueName: \"kubernetes.io/projected/e57a8569-877d-48ac-82e4-6c5d2ace9f99-kube-api-access-g6vpf\") pod \"e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:32.792139 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.792118 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e57a8569-877d-48ac-82e4-6c5d2ace9f99-istio-token\") pod \"e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:32.961723 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:32.961601 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:33.108599 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:33.108574 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf"] Jul 06 11:57:33.109793 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:57:33.109768 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode57a8569_877d_48ac_82e4_6c5d2ace9f99.slice/crio-ec873f7e535715bcb97287e712871ea00fc57cfb16734dbd108a630420b52bde WatchSource:0}: Error finding container ec873f7e535715bcb97287e712871ea00fc57cfb16734dbd108a630420b52bde: Status 404 returned error can't find the container with id ec873f7e535715bcb97287e712871ea00fc57cfb16734dbd108a630420b52bde Jul 06 11:57:33.112206 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:33.112170 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 11:57:33.112288 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:33.112242 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 11:57:33.112288 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:33.112277 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 11:57:33.738354 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:33.738310 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" event={"ID":"e57a8569-877d-48ac-82e4-6c5d2ace9f99","Type":"ContainerStarted","Data":"e6fa76f4c6a8ceffb2a15958709f63583bb4b23d12f1e7cce00175c5a4846194"} Jul 06 11:57:33.738774 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:33.738378 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" event={"ID":"e57a8569-877d-48ac-82e4-6c5d2ace9f99","Type":"ContainerStarted","Data":"ec873f7e535715bcb97287e712871ea00fc57cfb16734dbd108a630420b52bde"} Jul 06 11:57:33.784613 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:33.784566 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" podStartSLOduration=1.784549522 podStartE2EDuration="1.784549522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:57:33.783186972 +0000 UTC m=+1752.262985990" watchObservedRunningTime="2026-07-06 11:57:33.784549522 +0000 UTC m=+1752.264348538" Jul 06 11:57:33.961935 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:33.961900 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:34.967182 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:34.967150 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:35.750572 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:35.750543 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:35.751389 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:35.751370 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:35.936337 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:35.936299 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7"] Jul 06 11:57:35.940012 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:35.939990 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7" Jul 06 11:57:35.942794 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:35.942776 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-07d1fc4e-serving-cert\"" Jul 06 11:57:35.951908 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:35.951884 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7"] Jul 06 11:57:36.011916 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:36.011839 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed-maas-api-tls\") pod \"maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7\" (UID: \"b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed\") " pod="opendatahub/maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7" Jul 06 11:57:36.011916 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:36.011871 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed-openshift-service-ca\") pod \"maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7\" (UID: \"b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed\") " pod="opendatahub/maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7" Jul 06 11:57:36.012296 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:36.012006 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8wkf\" (UniqueName: \"kubernetes.io/projected/b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed-kube-api-access-p8wkf\") pod \"maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7\" (UID: \"b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed\") " pod="opendatahub/maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7" Jul 06 11:57:36.113416 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:36.113381 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8wkf\" (UniqueName: \"kubernetes.io/projected/b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed-kube-api-access-p8wkf\") pod \"maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7\" (UID: \"b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed\") " pod="opendatahub/maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7" Jul 06 11:57:36.113416 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:36.113421 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed-maas-api-tls\") pod \"maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7\" (UID: \"b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed\") " pod="opendatahub/maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7" Jul 06 11:57:36.113706 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:36.113438 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed-openshift-service-ca\") pod \"maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7\" (UID: \"b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed\") " pod="opendatahub/maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7" Jul 06 11:57:36.113706 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:57:36.113558 2568 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-07d1fc4e-serving-cert: secret "maas-api-e2e-mt-07d1fc4e-serving-cert" not found Jul 06 11:57:36.113706 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:57:36.113624 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed-maas-api-tls podName:b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed nodeName:}" failed. No retries permitted until 2026-07-06 11:57:36.613606759 +0000 UTC m=+1755.093405753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed-maas-api-tls") pod "maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7" (UID: "b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed") : secret "maas-api-e2e-mt-07d1fc4e-serving-cert" not found Jul 06 11:57:36.114062 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:36.114044 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed-openshift-service-ca\") pod \"maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7\" (UID: \"b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed\") " pod="opendatahub/maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7" Jul 06 11:57:36.126333 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:36.126311 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8wkf\" (UniqueName: \"kubernetes.io/projected/b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed-kube-api-access-p8wkf\") pod \"maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7\" (UID: \"b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed\") " pod="opendatahub/maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7" Jul 06 11:57:36.618442 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:36.618403 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed-maas-api-tls\") pod \"maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7\" (UID: \"b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed\") " pod="opendatahub/maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7" Jul 06 11:57:36.620813 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:36.620792 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed-maas-api-tls\") pod \"maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7\" (UID: \"b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed\") " pod="opendatahub/maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7" Jul 06 11:57:36.852026 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:36.851992 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7" Jul 06 11:57:36.991223 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:36.991188 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7"] Jul 06 11:57:36.992815 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:57:36.992790 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb53c3e9f_edcd_4bb8_ac0e_9db9c91175ed.slice/crio-5971815d05cc91ad025b1edf03cebd5493028acab219e84cf4e00f99957483ad WatchSource:0}: Error finding container 5971815d05cc91ad025b1edf03cebd5493028acab219e84cf4e00f99957483ad: Status 404 returned error can't find the container with id 5971815d05cc91ad025b1edf03cebd5493028acab219e84cf4e00f99957483ad Jul 06 11:57:37.762715 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:37.762600 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7" event={"ID":"b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed","Type":"ContainerStarted","Data":"7b0fcbacc255c5fb10f54bf27b22bbb434ed3131b1e0b3a3ca54e1d737110721"} Jul 06 11:57:37.762715 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:37.762655 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7" event={"ID":"b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed","Type":"ContainerStarted","Data":"5971815d05cc91ad025b1edf03cebd5493028acab219e84cf4e00f99957483ad"} Jul 06 11:57:37.763163 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:37.762723 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7" Jul 06 11:57:37.783397 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:37.783351 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7" podStartSLOduration=2.299310839 podStartE2EDuration="2.783336553s" totalImagesPullingTime="484.025714ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:57:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:57:37.782278794 +0000 UTC m=+1756.262077812" watchObservedRunningTime="2026-07-06 11:57:37.783336553 +0000 UTC m=+1756.263135569" Jul 06 11:57:43.774668 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:43.774639 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7" Jul 06 11:57:48.037705 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:48.035815 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7"] Jul 06 11:57:48.037705 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:48.036026 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7" podUID="b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed" containerName="maas-api" containerID="cri-o://7b0fcbacc255c5fb10f54bf27b22bbb434ed3131b1e0b3a3ca54e1d737110721" gracePeriod=30 Jul 06 11:57:48.286129 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:48.286105 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7" Jul 06 11:57:48.335314 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:48.330006 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8wkf\" (UniqueName: \"kubernetes.io/projected/b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed-kube-api-access-p8wkf\") pod \"b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed\" (UID: \"b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed\") " Jul 06 11:57:48.335314 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:48.330064 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed-openshift-service-ca\") pod \"b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed\" (UID: \"b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed\") " Jul 06 11:57:48.335314 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:48.330094 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed-maas-api-tls\") pod \"b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed\" (UID: \"b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed\") " Jul 06 11:57:48.335314 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:48.330667 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed" (UID: "b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 11:57:48.335314 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:48.333023 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed-kube-api-access-p8wkf" (OuterVolumeSpecName: "kube-api-access-p8wkf") pod "b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed" (UID: "b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed"). InnerVolumeSpecName "kube-api-access-p8wkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 11:57:48.335314 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:48.333342 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed" (UID: "b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 11:57:48.431805 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:48.431772 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8wkf\" (UniqueName: \"kubernetes.io/projected/b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed-kube-api-access-p8wkf\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:57:48.431937 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:48.431809 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed-openshift-service-ca\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:57:48.431937 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:48.431825 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed-maas-api-tls\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:57:48.822836 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:48.822805 2568 generic.go:356] "Generic (PLEG): container finished" podID="e57a8569-877d-48ac-82e4-6c5d2ace9f99" containerID="e6fa76f4c6a8ceffb2a15958709f63583bb4b23d12f1e7cce00175c5a4846194" exitCode=0 Jul 06 11:57:48.823015 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:48.822880 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" event={"ID":"e57a8569-877d-48ac-82e4-6c5d2ace9f99","Type":"ContainerDied","Data":"e6fa76f4c6a8ceffb2a15958709f63583bb4b23d12f1e7cce00175c5a4846194"} Jul 06 11:57:48.823351 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:48.823330 2568 scope.go:117] "RemoveContainer" containerID="e6fa76f4c6a8ceffb2a15958709f63583bb4b23d12f1e7cce00175c5a4846194" Jul 06 11:57:48.824189 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:48.824142 2568 generic.go:356] "Generic (PLEG): container finished" podID="b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed" containerID="7b0fcbacc255c5fb10f54bf27b22bbb434ed3131b1e0b3a3ca54e1d737110721" exitCode=0 Jul 06 11:57:48.824291 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:48.824214 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7" Jul 06 11:57:48.824291 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:48.824235 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7" event={"ID":"b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed","Type":"ContainerDied","Data":"7b0fcbacc255c5fb10f54bf27b22bbb434ed3131b1e0b3a3ca54e1d737110721"} Jul 06 11:57:48.824291 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:48.824286 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7" event={"ID":"b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed","Type":"ContainerDied","Data":"5971815d05cc91ad025b1edf03cebd5493028acab219e84cf4e00f99957483ad"} Jul 06 11:57:48.824415 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:48.824312 2568 scope.go:117] "RemoveContainer" containerID="7b0fcbacc255c5fb10f54bf27b22bbb434ed3131b1e0b3a3ca54e1d737110721" Jul 06 11:57:48.824558 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:48.824494 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 11:57:48.824649 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:48.824604 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 11:57:48.824732 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:48.824663 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 11:57:48.837604 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:48.837586 2568 scope.go:117] "RemoveContainer" containerID="7b0fcbacc255c5fb10f54bf27b22bbb434ed3131b1e0b3a3ca54e1d737110721" Jul 06 11:57:48.837884 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:57:48.837857 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b0fcbacc255c5fb10f54bf27b22bbb434ed3131b1e0b3a3ca54e1d737110721\": container with ID starting with 7b0fcbacc255c5fb10f54bf27b22bbb434ed3131b1e0b3a3ca54e1d737110721 not found: ID does not exist" containerID="7b0fcbacc255c5fb10f54bf27b22bbb434ed3131b1e0b3a3ca54e1d737110721" Jul 06 11:57:48.837962 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:48.837886 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b0fcbacc255c5fb10f54bf27b22bbb434ed3131b1e0b3a3ca54e1d737110721"} err="failed to get container status \"7b0fcbacc255c5fb10f54bf27b22bbb434ed3131b1e0b3a3ca54e1d737110721\": rpc error: code = NotFound desc = could not find container \"7b0fcbacc255c5fb10f54bf27b22bbb434ed3131b1e0b3a3ca54e1d737110721\": container with ID starting with 7b0fcbacc255c5fb10f54bf27b22bbb434ed3131b1e0b3a3ca54e1d737110721 not found: ID does not exist" Jul 06 11:57:48.875870 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:48.875845 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7"] Jul 06 11:57:48.883797 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:48.883776 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-07d1fc4e-6cf4ff4858-8rqw7"] Jul 06 11:57:49.832023 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:49.831978 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" event={"ID":"e57a8569-877d-48ac-82e4-6c5d2ace9f99","Type":"ContainerStarted","Data":"62af266aa58681d1f7ae8dd5711de1352ad50490057c7cbd5067a4f761e0a72b"} Jul 06 11:57:49.962799 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:49.962762 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:49.963888 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:49.963854 2568 patch_prober.go:28] interesting pod/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 06 11:57:49.964049 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:49.963911 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" podUID="e57a8569-877d-48ac-82e4-6c5d2ace9f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 06 11:57:50.056099 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:50.056045 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed" path="/var/lib/kubelet/pods/b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed/volumes" Jul 06 11:57:50.750779 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:50.750740 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:50.962666 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:50.962628 2568 patch_prober.go:28] interesting pod/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 06 11:57:50.963075 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:50.962717 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" podUID="e57a8569-877d-48ac-82e4-6c5d2ace9f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 06 11:57:51.962701 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:51.962647 2568 patch_prober.go:28] interesting pod/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 06 11:57:51.963097 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:51.962723 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" podUID="e57a8569-877d-48ac-82e4-6c5d2ace9f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 06 11:57:52.962131 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:52.962092 2568 patch_prober.go:28] interesting pod/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 06 11:57:52.962316 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:52.962149 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" podUID="e57a8569-877d-48ac-82e4-6c5d2ace9f99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 06 11:57:53.965911 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:53.965881 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:53.966886 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:53.966866 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:57:54.914524 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:54.914491 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf"] Jul 06 11:57:55.883863 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:57:55.883805 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" podUID="e57a8569-877d-48ac-82e4-6c5d2ace9f99" containerName="istio-proxy" containerID="cri-o://62af266aa58681d1f7ae8dd5711de1352ad50490057c7cbd5067a4f761e0a72b" gracePeriod=30 Jul 06 11:58:01.142094 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.142068 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:58:01.265234 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.265154 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57a8569-877d-48ac-82e4-6c5d2ace9f99-workload-socket\") pod \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " Jul 06 11:58:01.265234 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.265203 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e57a8569-877d-48ac-82e4-6c5d2ace9f99-istio-podinfo\") pod \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " Jul 06 11:58:01.265456 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.265239 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e57a8569-877d-48ac-82e4-6c5d2ace9f99-istio-token\") pod \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " Jul 06 11:58:01.265456 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.265279 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e57a8569-877d-48ac-82e4-6c5d2ace9f99-istio-envoy\") pod \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " Jul 06 11:58:01.265456 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.265335 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57a8569-877d-48ac-82e4-6c5d2ace9f99-credential-socket\") pod \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " Jul 06 11:58:01.265456 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.265385 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57a8569-877d-48ac-82e4-6c5d2ace9f99-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e57a8569-877d-48ac-82e4-6c5d2ace9f99" (UID: "e57a8569-877d-48ac-82e4-6c5d2ace9f99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:58:01.265456 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.265395 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6vpf\" (UniqueName: \"kubernetes.io/projected/e57a8569-877d-48ac-82e4-6c5d2ace9f99-kube-api-access-g6vpf\") pod \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " Jul 06 11:58:01.265456 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.265450 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e57a8569-877d-48ac-82e4-6c5d2ace9f99-istiod-ca-cert\") pod \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " Jul 06 11:58:01.265791 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.265493 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e57a8569-877d-48ac-82e4-6c5d2ace9f99-workload-certs\") pod \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " Jul 06 11:58:01.265791 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.265518 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e57a8569-877d-48ac-82e4-6c5d2ace9f99-istio-data\") pod \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\" (UID: \"e57a8569-877d-48ac-82e4-6c5d2ace9f99\") " Jul 06 11:58:01.265791 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.265670 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57a8569-877d-48ac-82e4-6c5d2ace9f99-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e57a8569-877d-48ac-82e4-6c5d2ace9f99" (UID: "e57a8569-877d-48ac-82e4-6c5d2ace9f99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:58:01.265986 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.265942 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e57a8569-877d-48ac-82e4-6c5d2ace9f99-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e57a8569-877d-48ac-82e4-6c5d2ace9f99" (UID: "e57a8569-877d-48ac-82e4-6c5d2ace9f99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 11:58:01.265986 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.265974 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57a8569-877d-48ac-82e4-6c5d2ace9f99-credential-socket\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:58:01.266272 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.265996 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57a8569-877d-48ac-82e4-6c5d2ace9f99-workload-socket\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:58:01.266272 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.266110 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57a8569-877d-48ac-82e4-6c5d2ace9f99-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e57a8569-877d-48ac-82e4-6c5d2ace9f99" (UID: "e57a8569-877d-48ac-82e4-6c5d2ace9f99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:58:01.266507 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.266481 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57a8569-877d-48ac-82e4-6c5d2ace9f99-istio-data" (OuterVolumeSpecName: "istio-data") pod "e57a8569-877d-48ac-82e4-6c5d2ace9f99" (UID: "e57a8569-877d-48ac-82e4-6c5d2ace9f99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:58:01.267471 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.267445 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57a8569-877d-48ac-82e4-6c5d2ace9f99-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e57a8569-877d-48ac-82e4-6c5d2ace9f99" (UID: "e57a8569-877d-48ac-82e4-6c5d2ace9f99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:58:01.267471 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.267452 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e57a8569-877d-48ac-82e4-6c5d2ace9f99-istio-token" (OuterVolumeSpecName: "istio-token") pod "e57a8569-877d-48ac-82e4-6c5d2ace9f99" (UID: "e57a8569-877d-48ac-82e4-6c5d2ace9f99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 11:58:01.267616 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.267489 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e57a8569-877d-48ac-82e4-6c5d2ace9f99-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e57a8569-877d-48ac-82e4-6c5d2ace9f99" (UID: "e57a8569-877d-48ac-82e4-6c5d2ace9f99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 11:58:01.267892 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.267876 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e57a8569-877d-48ac-82e4-6c5d2ace9f99-kube-api-access-g6vpf" (OuterVolumeSpecName: "kube-api-access-g6vpf") pod "e57a8569-877d-48ac-82e4-6c5d2ace9f99" (UID: "e57a8569-877d-48ac-82e4-6c5d2ace9f99"). InnerVolumeSpecName "kube-api-access-g6vpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 11:58:01.366556 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.366527 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e57a8569-877d-48ac-82e4-6c5d2ace9f99-istio-envoy\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:58:01.366556 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.366552 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6vpf\" (UniqueName: \"kubernetes.io/projected/e57a8569-877d-48ac-82e4-6c5d2ace9f99-kube-api-access-g6vpf\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:58:01.366556 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.366562 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e57a8569-877d-48ac-82e4-6c5d2ace9f99-istiod-ca-cert\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:58:01.366782 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.366570 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e57a8569-877d-48ac-82e4-6c5d2ace9f99-workload-certs\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:58:01.366782 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.366579 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e57a8569-877d-48ac-82e4-6c5d2ace9f99-istio-data\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:58:01.366782 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.366586 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e57a8569-877d-48ac-82e4-6c5d2ace9f99-istio-podinfo\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:58:01.366782 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.366593 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e57a8569-877d-48ac-82e4-6c5d2ace9f99-istio-token\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:58:01.920338 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.920299 2568 generic.go:356] "Generic (PLEG): container finished" podID="e57a8569-877d-48ac-82e4-6c5d2ace9f99" containerID="62af266aa58681d1f7ae8dd5711de1352ad50490057c7cbd5067a4f761e0a72b" exitCode=0 Jul 06 11:58:01.920524 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.920364 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" Jul 06 11:58:01.920524 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.920403 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" event={"ID":"e57a8569-877d-48ac-82e4-6c5d2ace9f99","Type":"ContainerDied","Data":"62af266aa58681d1f7ae8dd5711de1352ad50490057c7cbd5067a4f761e0a72b"} Jul 06 11:58:01.920524 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.920455 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf" event={"ID":"e57a8569-877d-48ac-82e4-6c5d2ace9f99","Type":"ContainerDied","Data":"ec873f7e535715bcb97287e712871ea00fc57cfb16734dbd108a630420b52bde"} Jul 06 11:58:01.920746 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.920552 2568 scope.go:117] "RemoveContainer" containerID="62af266aa58681d1f7ae8dd5711de1352ad50490057c7cbd5067a4f761e0a72b" Jul 06 11:58:01.932106 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.932084 2568 scope.go:117] "RemoveContainer" containerID="e6fa76f4c6a8ceffb2a15958709f63583bb4b23d12f1e7cce00175c5a4846194" Jul 06 11:58:01.940490 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.940471 2568 scope.go:117] "RemoveContainer" containerID="62af266aa58681d1f7ae8dd5711de1352ad50490057c7cbd5067a4f761e0a72b" Jul 06 11:58:01.940786 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:58:01.940766 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62af266aa58681d1f7ae8dd5711de1352ad50490057c7cbd5067a4f761e0a72b\": container with ID starting with 62af266aa58681d1f7ae8dd5711de1352ad50490057c7cbd5067a4f761e0a72b not found: ID does not exist" containerID="62af266aa58681d1f7ae8dd5711de1352ad50490057c7cbd5067a4f761e0a72b" Jul 06 11:58:01.940880 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.940800 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62af266aa58681d1f7ae8dd5711de1352ad50490057c7cbd5067a4f761e0a72b"} err="failed to get container status \"62af266aa58681d1f7ae8dd5711de1352ad50490057c7cbd5067a4f761e0a72b\": rpc error: code = NotFound desc = could not find container \"62af266aa58681d1f7ae8dd5711de1352ad50490057c7cbd5067a4f761e0a72b\": container with ID starting with 62af266aa58681d1f7ae8dd5711de1352ad50490057c7cbd5067a4f761e0a72b not found: ID does not exist" Jul 06 11:58:01.940880 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.940828 2568 scope.go:117] "RemoveContainer" containerID="e6fa76f4c6a8ceffb2a15958709f63583bb4b23d12f1e7cce00175c5a4846194" Jul 06 11:58:01.941125 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:58:01.941106 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6fa76f4c6a8ceffb2a15958709f63583bb4b23d12f1e7cce00175c5a4846194\": container with ID starting with e6fa76f4c6a8ceffb2a15958709f63583bb4b23d12f1e7cce00175c5a4846194 not found: ID does not exist" containerID="e6fa76f4c6a8ceffb2a15958709f63583bb4b23d12f1e7cce00175c5a4846194" Jul 06 11:58:01.941161 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.941131 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6fa76f4c6a8ceffb2a15958709f63583bb4b23d12f1e7cce00175c5a4846194"} err="failed to get container status \"e6fa76f4c6a8ceffb2a15958709f63583bb4b23d12f1e7cce00175c5a4846194\": rpc error: code = NotFound desc = could not find container \"e6fa76f4c6a8ceffb2a15958709f63583bb4b23d12f1e7cce00175c5a4846194\": container with ID starting with e6fa76f4c6a8ceffb2a15958709f63583bb4b23d12f1e7cce00175c5a4846194 not found: ID does not exist" Jul 06 11:58:01.984034 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.984001 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf"] Jul 06 11:58:01.998366 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:01.998299 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-07d1fc4e-openshift-default-64846dd9f5-kjgjf"] Jul 06 11:58:02.052321 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:02.052288 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e57a8569-877d-48ac-82e4-6c5d2ace9f99" path="/var/lib/kubelet/pods/e57a8569-877d-48ac-82e4-6c5d2ace9f99/volumes" Jul 06 11:58:14.873727 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:14.873628 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf"] Jul 06 11:58:14.874224 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:14.874160 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e57a8569-877d-48ac-82e4-6c5d2ace9f99" containerName="istio-proxy" Jul 06 11:58:14.874224 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:14.874179 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e57a8569-877d-48ac-82e4-6c5d2ace9f99" containerName="istio-proxy" Jul 06 11:58:14.874224 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:14.874197 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed" containerName="maas-api" Jul 06 11:58:14.874224 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:14.874205 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed" containerName="maas-api" Jul 06 11:58:14.874431 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:14.874316 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b53c3e9f-edcd-4bb8-ac0e-9db9c91175ed" containerName="maas-api" Jul 06 11:58:14.874431 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:14.874332 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e57a8569-877d-48ac-82e4-6c5d2ace9f99" containerName="istio-proxy" Jul 06 11:58:14.874431 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:14.874343 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e57a8569-877d-48ac-82e4-6c5d2ace9f99" containerName="istio-proxy" Jul 06 11:58:14.879061 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:14.879035 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:14.881869 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:14.881844 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b3137601-openshift-default-dockercfg-m7kbt\"" Jul 06 11:58:14.891605 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:14.891577 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf"] Jul 06 11:58:14.983095 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:14.983048 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/651d6193-a006-4619-82d5-89cee07a11b7-credential-socket\") pod \"e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:14.983281 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:14.983102 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/651d6193-a006-4619-82d5-89cee07a11b7-istiod-ca-cert\") pod \"e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:14.983281 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:14.983235 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/651d6193-a006-4619-82d5-89cee07a11b7-istio-envoy\") pod \"e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:14.983281 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:14.983273 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/651d6193-a006-4619-82d5-89cee07a11b7-workload-socket\") pod \"e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:14.983554 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:14.983311 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/651d6193-a006-4619-82d5-89cee07a11b7-istio-token\") pod \"e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:14.983554 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:14.983351 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/651d6193-a006-4619-82d5-89cee07a11b7-istio-data\") pod \"e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:14.983554 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:14.983409 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/651d6193-a006-4619-82d5-89cee07a11b7-workload-certs\") pod \"e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:14.983554 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:14.983440 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/651d6193-a006-4619-82d5-89cee07a11b7-istio-podinfo\") pod \"e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:14.983554 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:14.983464 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjsl2\" (UniqueName: \"kubernetes.io/projected/651d6193-a006-4619-82d5-89cee07a11b7-kube-api-access-rjsl2\") pod \"e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:15.084439 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:15.084395 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/651d6193-a006-4619-82d5-89cee07a11b7-workload-certs\") pod \"e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:15.084439 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:15.084445 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/651d6193-a006-4619-82d5-89cee07a11b7-istio-podinfo\") pod \"e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:15.084720 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:15.084470 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjsl2\" (UniqueName: \"kubernetes.io/projected/651d6193-a006-4619-82d5-89cee07a11b7-kube-api-access-rjsl2\") pod \"e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:15.084720 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:15.084503 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/651d6193-a006-4619-82d5-89cee07a11b7-credential-socket\") pod \"e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:15.084720 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:15.084532 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/651d6193-a006-4619-82d5-89cee07a11b7-istiod-ca-cert\") pod \"e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:15.084720 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:15.084581 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/651d6193-a006-4619-82d5-89cee07a11b7-istio-envoy\") pod \"e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:15.084720 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:15.084601 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/651d6193-a006-4619-82d5-89cee07a11b7-workload-socket\") pod \"e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:15.084720 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:15.084624 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/651d6193-a006-4619-82d5-89cee07a11b7-istio-token\") pod \"e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:15.084720 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:15.084663 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/651d6193-a006-4619-82d5-89cee07a11b7-istio-data\") pod \"e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:15.085105 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:15.084936 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/651d6193-a006-4619-82d5-89cee07a11b7-credential-socket\") pod \"e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:15.085105 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:15.085033 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/651d6193-a006-4619-82d5-89cee07a11b7-workload-certs\") pod \"e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:15.085223 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:15.085152 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/651d6193-a006-4619-82d5-89cee07a11b7-workload-socket\") pod \"e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:15.085223 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:15.085177 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/651d6193-a006-4619-82d5-89cee07a11b7-istio-data\") pod \"e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:15.085401 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:15.085379 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/651d6193-a006-4619-82d5-89cee07a11b7-istiod-ca-cert\") pod \"e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:15.086881 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:15.086864 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/651d6193-a006-4619-82d5-89cee07a11b7-istio-envoy\") pod \"e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:15.087160 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:15.087145 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/651d6193-a006-4619-82d5-89cee07a11b7-istio-podinfo\") pod \"e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:15.093650 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:15.093616 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/651d6193-a006-4619-82d5-89cee07a11b7-istio-token\") pod \"e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:15.093843 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:15.093821 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjsl2\" (UniqueName: \"kubernetes.io/projected/651d6193-a006-4619-82d5-89cee07a11b7-kube-api-access-rjsl2\") pod \"e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:15.190822 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:15.190786 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:15.334688 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:15.334641 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf"] Jul 06 11:58:15.336238 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:58:15.336209 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod651d6193_a006_4619_82d5_89cee07a11b7.slice/crio-639d088807134dff664d437608e7225626cf6a987606b395d89fd53bd8c3d05d WatchSource:0}: Error finding container 639d088807134dff664d437608e7225626cf6a987606b395d89fd53bd8c3d05d: Status 404 returned error can't find the container with id 639d088807134dff664d437608e7225626cf6a987606b395d89fd53bd8c3d05d Jul 06 11:58:15.338418 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:15.338381 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 11:58:15.338547 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:15.338472 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 11:58:15.338547 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:15.338516 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 11:58:16.000967 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:16.000919 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" event={"ID":"651d6193-a006-4619-82d5-89cee07a11b7","Type":"ContainerStarted","Data":"b31939642276831b53c39b3c2faa782147dc61d340d0d17ebf95e785366d9691"} Jul 06 11:58:16.001419 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:16.000993 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" event={"ID":"651d6193-a006-4619-82d5-89cee07a11b7","Type":"ContainerStarted","Data":"639d088807134dff664d437608e7225626cf6a987606b395d89fd53bd8c3d05d"} Jul 06 11:58:16.032698 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:16.032622 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" podStartSLOduration=2.032606276 podStartE2EDuration="2.032606276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:58:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:58:16.030150914 +0000 UTC m=+1794.509949933" watchObservedRunningTime="2026-07-06 11:58:16.032606276 +0000 UTC m=+1794.512405291" Jul 06 11:58:16.191861 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:16.191829 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:16.197106 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:16.197081 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:17.006409 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:17.006381 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:17.007444 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:17.007423 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:18.089794 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:18.089758 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx"] Jul 06 11:58:18.090185 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:18.090117 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e57a8569-877d-48ac-82e4-6c5d2ace9f99" containerName="istio-proxy" Jul 06 11:58:18.090185 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:18.090128 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e57a8569-877d-48ac-82e4-6c5d2ace9f99" containerName="istio-proxy" Jul 06 11:58:18.093274 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:18.093254 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx" Jul 06 11:58:18.095846 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:18.095818 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-b3137601-serving-cert\"" Jul 06 11:58:18.103014 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:18.102990 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx"] Jul 06 11:58:18.111899 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:18.111869 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flmdr\" (UniqueName: \"kubernetes.io/projected/a822f3ec-acf3-4e40-91c7-57d0c832abf3-kube-api-access-flmdr\") pod \"maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx\" (UID: \"a822f3ec-acf3-4e40-91c7-57d0c832abf3\") " pod="opendatahub/maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx" Jul 06 11:58:18.112209 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:18.112185 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a822f3ec-acf3-4e40-91c7-57d0c832abf3-openshift-service-ca\") pod \"maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx\" (UID: \"a822f3ec-acf3-4e40-91c7-57d0c832abf3\") " pod="opendatahub/maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx" Jul 06 11:58:18.112408 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:18.112374 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a822f3ec-acf3-4e40-91c7-57d0c832abf3-maas-api-tls\") pod \"maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx\" (UID: \"a822f3ec-acf3-4e40-91c7-57d0c832abf3\") " pod="opendatahub/maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx" Jul 06 11:58:18.213319 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:18.213280 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flmdr\" (UniqueName: \"kubernetes.io/projected/a822f3ec-acf3-4e40-91c7-57d0c832abf3-kube-api-access-flmdr\") pod \"maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx\" (UID: \"a822f3ec-acf3-4e40-91c7-57d0c832abf3\") " pod="opendatahub/maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx" Jul 06 11:58:18.213486 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:18.213342 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a822f3ec-acf3-4e40-91c7-57d0c832abf3-openshift-service-ca\") pod \"maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx\" (UID: \"a822f3ec-acf3-4e40-91c7-57d0c832abf3\") " pod="opendatahub/maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx" Jul 06 11:58:18.213486 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:18.213383 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a822f3ec-acf3-4e40-91c7-57d0c832abf3-maas-api-tls\") pod \"maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx\" (UID: \"a822f3ec-acf3-4e40-91c7-57d0c832abf3\") " pod="opendatahub/maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx" Jul 06 11:58:18.213554 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:58:18.213515 2568 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-b3137601-serving-cert: secret "maas-api-e2e-mt-b3137601-serving-cert" not found Jul 06 11:58:18.213606 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:58:18.213573 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a822f3ec-acf3-4e40-91c7-57d0c832abf3-maas-api-tls podName:a822f3ec-acf3-4e40-91c7-57d0c832abf3 nodeName:}" failed. No retries permitted until 2026-07-06 11:58:18.713554796 +0000 UTC m=+1797.193353795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a822f3ec-acf3-4e40-91c7-57d0c832abf3-maas-api-tls") pod "maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx" (UID: "a822f3ec-acf3-4e40-91c7-57d0c832abf3") : secret "maas-api-e2e-mt-b3137601-serving-cert" not found Jul 06 11:58:18.214019 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:18.214001 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a822f3ec-acf3-4e40-91c7-57d0c832abf3-openshift-service-ca\") pod \"maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx\" (UID: \"a822f3ec-acf3-4e40-91c7-57d0c832abf3\") " pod="opendatahub/maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx" Jul 06 11:58:18.242479 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:18.242448 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flmdr\" (UniqueName: \"kubernetes.io/projected/a822f3ec-acf3-4e40-91c7-57d0c832abf3-kube-api-access-flmdr\") pod \"maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx\" (UID: \"a822f3ec-acf3-4e40-91c7-57d0c832abf3\") " pod="opendatahub/maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx" Jul 06 11:58:18.718258 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:18.718221 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a822f3ec-acf3-4e40-91c7-57d0c832abf3-maas-api-tls\") pod \"maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx\" (UID: \"a822f3ec-acf3-4e40-91c7-57d0c832abf3\") " pod="opendatahub/maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx" Jul 06 11:58:18.721176 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:18.721151 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a822f3ec-acf3-4e40-91c7-57d0c832abf3-maas-api-tls\") pod \"maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx\" (UID: \"a822f3ec-acf3-4e40-91c7-57d0c832abf3\") " pod="opendatahub/maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx" Jul 06 11:58:19.006446 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:19.006360 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx" Jul 06 11:58:19.151189 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:19.151162 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx"] Jul 06 11:58:19.152714 ip-10-0-131-176 kubenswrapper[2568]: W0706 11:58:19.152686 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda822f3ec_acf3_4e40_91c7_57d0c832abf3.slice/crio-1e8befef03cdba9f7e451b9e1b0e858c01e5fe21a19a895ef931d7f3af27ecb4 WatchSource:0}: Error finding container 1e8befef03cdba9f7e451b9e1b0e858c01e5fe21a19a895ef931d7f3af27ecb4: Status 404 returned error can't find the container with id 1e8befef03cdba9f7e451b9e1b0e858c01e5fe21a19a895ef931d7f3af27ecb4 Jul 06 11:58:20.025691 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:20.025648 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx" event={"ID":"a822f3ec-acf3-4e40-91c7-57d0c832abf3","Type":"ContainerStarted","Data":"7b23b05a4eafe4eab291b15dc03a1e8513071e77ad2b9c8e9b5b19e0aec5122e"} Jul 06 11:58:20.025909 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:20.025736 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx" event={"ID":"a822f3ec-acf3-4e40-91c7-57d0c832abf3","Type":"ContainerStarted","Data":"1e8befef03cdba9f7e451b9e1b0e858c01e5fe21a19a895ef931d7f3af27ecb4"} Jul 06 11:58:20.025909 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:20.025762 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx" Jul 06 11:58:20.045252 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:20.045196 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx" podStartSLOduration=1.7256528709999999 podStartE2EDuration="2.045176296s" totalImagesPullingTime="319.523425ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 11:58:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 11:58:20.043059649 +0000 UTC m=+1798.522858682" watchObservedRunningTime="2026-07-06 11:58:20.045176296 +0000 UTC m=+1798.524975311" Jul 06 11:58:26.057183 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:26.057150 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx" Jul 06 11:58:32.008630 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:32.008591 2568 patch_prober.go:28] interesting pod/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 06 11:58:32.009192 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:32.008654 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" podUID="651d6193-a006-4619-82d5-89cee07a11b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 06 11:58:32.100476 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:32.100442 2568 generic.go:356] "Generic (PLEG): container finished" podID="651d6193-a006-4619-82d5-89cee07a11b7" containerID="b31939642276831b53c39b3c2faa782147dc61d340d0d17ebf95e785366d9691" exitCode=0 Jul 06 11:58:32.100714 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:32.100523 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" event={"ID":"651d6193-a006-4619-82d5-89cee07a11b7","Type":"ContainerDied","Data":"b31939642276831b53c39b3c2faa782147dc61d340d0d17ebf95e785366d9691"} Jul 06 11:58:32.101066 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:32.101050 2568 scope.go:117] "RemoveContainer" containerID="b31939642276831b53c39b3c2faa782147dc61d340d0d17ebf95e785366d9691" Jul 06 11:58:32.102107 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:32.102071 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 11:58:32.102205 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:32.102152 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 11:58:32.102205 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:32.102189 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 11:58:33.108379 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:33.108341 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" event={"ID":"651d6193-a006-4619-82d5-89cee07a11b7","Type":"ContainerStarted","Data":"a7a815178625e3e2b40663edd09d178988bbcd3417ce72941005ea2edaa1558c"} Jul 06 11:58:33.191889 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:33.191847 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:33.192097 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:33.192074 2568 patch_prober.go:28] interesting pod/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 06 11:58:33.192167 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:33.192134 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" podUID="651d6193-a006-4619-82d5-89cee07a11b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 06 11:58:34.192288 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:34.192244 2568 patch_prober.go:28] interesting pod/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 06 11:58:34.192754 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:34.192311 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" podUID="651d6193-a006-4619-82d5-89cee07a11b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 06 11:58:35.192058 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:35.192019 2568 patch_prober.go:28] interesting pod/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 06 11:58:35.192336 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:35.192078 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" podUID="651d6193-a006-4619-82d5-89cee07a11b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 06 11:58:36.192130 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:36.192088 2568 patch_prober.go:28] interesting pod/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 06 11:58:36.192331 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:36.192154 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" podUID="651d6193-a006-4619-82d5-89cee07a11b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 06 11:58:37.203707 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:37.203651 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:37.204278 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:37.204255 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:37.205342 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:37.205314 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:58:55.183864 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:55.183825 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx"] Jul 06 11:58:55.184402 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:55.184149 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx" podUID="a822f3ec-acf3-4e40-91c7-57d0c832abf3" containerName="maas-api" containerID="cri-o://7b23b05a4eafe4eab291b15dc03a1e8513071e77ad2b9c8e9b5b19e0aec5122e" gracePeriod=30 Jul 06 11:58:55.433360 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:55.433333 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx" Jul 06 11:58:55.569651 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:55.569547 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a822f3ec-acf3-4e40-91c7-57d0c832abf3-maas-api-tls\") pod \"a822f3ec-acf3-4e40-91c7-57d0c832abf3\" (UID: \"a822f3ec-acf3-4e40-91c7-57d0c832abf3\") " Jul 06 11:58:55.569841 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:55.569801 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a822f3ec-acf3-4e40-91c7-57d0c832abf3-openshift-service-ca\") pod \"a822f3ec-acf3-4e40-91c7-57d0c832abf3\" (UID: \"a822f3ec-acf3-4e40-91c7-57d0c832abf3\") " Jul 06 11:58:55.569896 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:55.569861 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flmdr\" (UniqueName: \"kubernetes.io/projected/a822f3ec-acf3-4e40-91c7-57d0c832abf3-kube-api-access-flmdr\") pod \"a822f3ec-acf3-4e40-91c7-57d0c832abf3\" (UID: \"a822f3ec-acf3-4e40-91c7-57d0c832abf3\") " Jul 06 11:58:55.570172 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:55.570141 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a822f3ec-acf3-4e40-91c7-57d0c832abf3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a822f3ec-acf3-4e40-91c7-57d0c832abf3" (UID: "a822f3ec-acf3-4e40-91c7-57d0c832abf3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 11:58:55.571893 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:55.571869 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a822f3ec-acf3-4e40-91c7-57d0c832abf3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a822f3ec-acf3-4e40-91c7-57d0c832abf3" (UID: "a822f3ec-acf3-4e40-91c7-57d0c832abf3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 11:58:55.571893 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:55.571887 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a822f3ec-acf3-4e40-91c7-57d0c832abf3-kube-api-access-flmdr" (OuterVolumeSpecName: "kube-api-access-flmdr") pod "a822f3ec-acf3-4e40-91c7-57d0c832abf3" (UID: "a822f3ec-acf3-4e40-91c7-57d0c832abf3"). InnerVolumeSpecName "kube-api-access-flmdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 11:58:55.670763 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:55.670724 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a822f3ec-acf3-4e40-91c7-57d0c832abf3-openshift-service-ca\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:58:55.670763 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:55.670756 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flmdr\" (UniqueName: \"kubernetes.io/projected/a822f3ec-acf3-4e40-91c7-57d0c832abf3-kube-api-access-flmdr\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:58:55.670763 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:55.670769 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a822f3ec-acf3-4e40-91c7-57d0c832abf3-maas-api-tls\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:58:56.239246 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:56.239214 2568 generic.go:356] "Generic (PLEG): container finished" podID="a822f3ec-acf3-4e40-91c7-57d0c832abf3" containerID="7b23b05a4eafe4eab291b15dc03a1e8513071e77ad2b9c8e9b5b19e0aec5122e" exitCode=0 Jul 06 11:58:56.239692 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:56.239259 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx" Jul 06 11:58:56.239692 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:56.239300 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx" event={"ID":"a822f3ec-acf3-4e40-91c7-57d0c832abf3","Type":"ContainerDied","Data":"7b23b05a4eafe4eab291b15dc03a1e8513071e77ad2b9c8e9b5b19e0aec5122e"} Jul 06 11:58:56.239692 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:56.239361 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx" event={"ID":"a822f3ec-acf3-4e40-91c7-57d0c832abf3","Type":"ContainerDied","Data":"1e8befef03cdba9f7e451b9e1b0e858c01e5fe21a19a895ef931d7f3af27ecb4"} Jul 06 11:58:56.239692 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:56.239385 2568 scope.go:117] "RemoveContainer" containerID="7b23b05a4eafe4eab291b15dc03a1e8513071e77ad2b9c8e9b5b19e0aec5122e" Jul 06 11:58:56.249028 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:56.249008 2568 scope.go:117] "RemoveContainer" containerID="7b23b05a4eafe4eab291b15dc03a1e8513071e77ad2b9c8e9b5b19e0aec5122e" Jul 06 11:58:56.249325 ip-10-0-131-176 kubenswrapper[2568]: E0706 11:58:56.249296 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b23b05a4eafe4eab291b15dc03a1e8513071e77ad2b9c8e9b5b19e0aec5122e\": container with ID starting with 7b23b05a4eafe4eab291b15dc03a1e8513071e77ad2b9c8e9b5b19e0aec5122e not found: ID does not exist" containerID="7b23b05a4eafe4eab291b15dc03a1e8513071e77ad2b9c8e9b5b19e0aec5122e" Jul 06 11:58:56.249413 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:56.249332 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b23b05a4eafe4eab291b15dc03a1e8513071e77ad2b9c8e9b5b19e0aec5122e"} err="failed to get container status \"7b23b05a4eafe4eab291b15dc03a1e8513071e77ad2b9c8e9b5b19e0aec5122e\": rpc error: code = NotFound desc = could not find container \"7b23b05a4eafe4eab291b15dc03a1e8513071e77ad2b9c8e9b5b19e0aec5122e\": container with ID starting with 7b23b05a4eafe4eab291b15dc03a1e8513071e77ad2b9c8e9b5b19e0aec5122e not found: ID does not exist" Jul 06 11:58:56.257850 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:56.257825 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx"] Jul 06 11:58:56.261592 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:56.261571 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-b3137601-7b6cbdfcb7-6b5qx"] Jul 06 11:58:58.052489 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:58:58.052443 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a822f3ec-acf3-4e40-91c7-57d0c832abf3" path="/var/lib/kubelet/pods/a822f3ec-acf3-4e40-91c7-57d0c832abf3/volumes" Jul 06 11:59:49.436092 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:49.435999 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf"] Jul 06 11:59:49.436620 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:49.436259 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" podUID="651d6193-a006-4619-82d5-89cee07a11b7" containerName="istio-proxy" containerID="cri-o://a7a815178625e3e2b40663edd09d178988bbcd3417ce72941005ea2edaa1558c" gracePeriod=30 Jul 06 11:59:52.206457 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:52.206417 2568 patch_prober.go:28] interesting pod/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 11:59:52.206892 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:52.206488 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" podUID="651d6193-a006-4619-82d5-89cee07a11b7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 11:59:54.555136 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:54.555106 2568 generic.go:356] "Generic (PLEG): container finished" podID="651d6193-a006-4619-82d5-89cee07a11b7" containerID="a7a815178625e3e2b40663edd09d178988bbcd3417ce72941005ea2edaa1558c" exitCode=0 Jul 06 11:59:54.555497 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:54.555181 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" event={"ID":"651d6193-a006-4619-82d5-89cee07a11b7","Type":"ContainerDied","Data":"a7a815178625e3e2b40663edd09d178988bbcd3417ce72941005ea2edaa1558c"} Jul 06 11:59:54.555497 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:54.555248 2568 scope.go:117] "RemoveContainer" containerID="b31939642276831b53c39b3c2faa782147dc61d340d0d17ebf95e785366d9691" Jul 06 11:59:54.700949 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:54.700880 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:59:54.813772 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:54.813739 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/651d6193-a006-4619-82d5-89cee07a11b7-istiod-ca-cert\") pod \"651d6193-a006-4619-82d5-89cee07a11b7\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " Jul 06 11:59:54.813772 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:54.813781 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/651d6193-a006-4619-82d5-89cee07a11b7-istio-data\") pod \"651d6193-a006-4619-82d5-89cee07a11b7\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " Jul 06 11:59:54.814038 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:54.813799 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/651d6193-a006-4619-82d5-89cee07a11b7-istio-podinfo\") pod \"651d6193-a006-4619-82d5-89cee07a11b7\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " Jul 06 11:59:54.814038 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:54.813824 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjsl2\" (UniqueName: \"kubernetes.io/projected/651d6193-a006-4619-82d5-89cee07a11b7-kube-api-access-rjsl2\") pod \"651d6193-a006-4619-82d5-89cee07a11b7\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " Jul 06 11:59:54.814038 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:54.813847 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/651d6193-a006-4619-82d5-89cee07a11b7-istio-token\") pod \"651d6193-a006-4619-82d5-89cee07a11b7\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " Jul 06 11:59:54.814038 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:54.813872 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/651d6193-a006-4619-82d5-89cee07a11b7-workload-certs\") pod \"651d6193-a006-4619-82d5-89cee07a11b7\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " Jul 06 11:59:54.814038 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:54.813892 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/651d6193-a006-4619-82d5-89cee07a11b7-istio-envoy\") pod \"651d6193-a006-4619-82d5-89cee07a11b7\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " Jul 06 11:59:54.814038 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:54.813912 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/651d6193-a006-4619-82d5-89cee07a11b7-credential-socket\") pod \"651d6193-a006-4619-82d5-89cee07a11b7\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " Jul 06 11:59:54.814038 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:54.813937 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/651d6193-a006-4619-82d5-89cee07a11b7-workload-socket\") pod \"651d6193-a006-4619-82d5-89cee07a11b7\" (UID: \"651d6193-a006-4619-82d5-89cee07a11b7\") " Jul 06 11:59:54.814415 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:54.814280 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651d6193-a006-4619-82d5-89cee07a11b7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "651d6193-a006-4619-82d5-89cee07a11b7" (UID: "651d6193-a006-4619-82d5-89cee07a11b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:59:54.814415 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:54.814289 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/651d6193-a006-4619-82d5-89cee07a11b7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "651d6193-a006-4619-82d5-89cee07a11b7" (UID: "651d6193-a006-4619-82d5-89cee07a11b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 11:59:54.814522 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:54.814486 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651d6193-a006-4619-82d5-89cee07a11b7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "651d6193-a006-4619-82d5-89cee07a11b7" (UID: "651d6193-a006-4619-82d5-89cee07a11b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:59:54.814522 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:54.814508 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651d6193-a006-4619-82d5-89cee07a11b7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "651d6193-a006-4619-82d5-89cee07a11b7" (UID: "651d6193-a006-4619-82d5-89cee07a11b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:59:54.814766 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:54.814734 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651d6193-a006-4619-82d5-89cee07a11b7-istio-data" (OuterVolumeSpecName: "istio-data") pod "651d6193-a006-4619-82d5-89cee07a11b7" (UID: "651d6193-a006-4619-82d5-89cee07a11b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:59:54.816221 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:54.816191 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/651d6193-a006-4619-82d5-89cee07a11b7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "651d6193-a006-4619-82d5-89cee07a11b7" (UID: "651d6193-a006-4619-82d5-89cee07a11b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 11:59:54.816221 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:54.816194 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/651d6193-a006-4619-82d5-89cee07a11b7-istio-token" (OuterVolumeSpecName: "istio-token") pod "651d6193-a006-4619-82d5-89cee07a11b7" (UID: "651d6193-a006-4619-82d5-89cee07a11b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 11:59:54.816221 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:54.816200 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/651d6193-a006-4619-82d5-89cee07a11b7-kube-api-access-rjsl2" (OuterVolumeSpecName: "kube-api-access-rjsl2") pod "651d6193-a006-4619-82d5-89cee07a11b7" (UID: "651d6193-a006-4619-82d5-89cee07a11b7"). InnerVolumeSpecName "kube-api-access-rjsl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 11:59:54.816398 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:54.816242 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651d6193-a006-4619-82d5-89cee07a11b7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "651d6193-a006-4619-82d5-89cee07a11b7" (UID: "651d6193-a006-4619-82d5-89cee07a11b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 11:59:54.914979 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:54.914940 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/651d6193-a006-4619-82d5-89cee07a11b7-istiod-ca-cert\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:59:54.914979 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:54.914973 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/651d6193-a006-4619-82d5-89cee07a11b7-istio-data\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:59:54.914979 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:54.914982 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/651d6193-a006-4619-82d5-89cee07a11b7-istio-podinfo\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:59:54.915232 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:54.914992 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjsl2\" (UniqueName: \"kubernetes.io/projected/651d6193-a006-4619-82d5-89cee07a11b7-kube-api-access-rjsl2\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:59:54.915232 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:54.915003 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/651d6193-a006-4619-82d5-89cee07a11b7-istio-token\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:59:54.915232 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:54.915012 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/651d6193-a006-4619-82d5-89cee07a11b7-workload-certs\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:59:54.915232 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:54.915019 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/651d6193-a006-4619-82d5-89cee07a11b7-istio-envoy\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:59:54.915232 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:54.915027 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/651d6193-a006-4619-82d5-89cee07a11b7-credential-socket\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:59:54.915232 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:54.915035 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/651d6193-a006-4619-82d5-89cee07a11b7-workload-socket\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 11:59:55.562511 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:55.562476 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" Jul 06 11:59:55.562511 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:55.562489 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf" event={"ID":"651d6193-a006-4619-82d5-89cee07a11b7","Type":"ContainerDied","Data":"639d088807134dff664d437608e7225626cf6a987606b395d89fd53bd8c3d05d"} Jul 06 11:59:55.563036 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:55.562554 2568 scope.go:117] "RemoveContainer" containerID="a7a815178625e3e2b40663edd09d178988bbcd3417ce72941005ea2edaa1558c" Jul 06 11:59:55.585453 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:55.585423 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf"] Jul 06 11:59:55.594096 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:55.594061 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b3137601-openshift-default-5cdb5454dc-brspf"] Jul 06 11:59:56.052616 ip-10-0-131-176 kubenswrapper[2568]: I0706 11:59:56.052580 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="651d6193-a006-4619-82d5-89cee07a11b7" path="/var/lib/kubelet/pods/651d6193-a006-4619-82d5-89cee07a11b7/volumes" Jul 06 12:00:00.142619 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:00.142572 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722320-rlln9"] Jul 06 12:00:00.143136 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:00.143119 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a822f3ec-acf3-4e40-91c7-57d0c832abf3" containerName="maas-api" Jul 06 12:00:00.143181 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:00.143140 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a822f3ec-acf3-4e40-91c7-57d0c832abf3" containerName="maas-api" Jul 06 12:00:00.143181 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:00.143162 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="651d6193-a006-4619-82d5-89cee07a11b7" containerName="istio-proxy" Jul 06 12:00:00.143181 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:00.143171 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="651d6193-a006-4619-82d5-89cee07a11b7" containerName="istio-proxy" Jul 06 12:00:00.143274 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:00.143183 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="651d6193-a006-4619-82d5-89cee07a11b7" containerName="istio-proxy" Jul 06 12:00:00.143274 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:00.143193 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="651d6193-a006-4619-82d5-89cee07a11b7" containerName="istio-proxy" Jul 06 12:00:00.143340 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:00.143280 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="651d6193-a006-4619-82d5-89cee07a11b7" containerName="istio-proxy" Jul 06 12:00:00.143340 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:00.143292 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a822f3ec-acf3-4e40-91c7-57d0c832abf3" containerName="maas-api" Jul 06 12:00:00.143340 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:00.143301 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="651d6193-a006-4619-82d5-89cee07a11b7" containerName="istio-proxy" Jul 06 12:00:00.147864 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:00.147840 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722320-rlln9" Jul 06 12:00:00.152423 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:00.152391 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722320-rlln9"] Jul 06 12:00:00.264042 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:00.263997 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp67m\" (UniqueName: \"kubernetes.io/projected/bf2357b0-6395-4bb3-8bfd-6099d105a418-kube-api-access-dp67m\") pod \"maas-api-key-cleanup-29722320-rlln9\" (UID: \"bf2357b0-6395-4bb3-8bfd-6099d105a418\") " pod="opendatahub/maas-api-key-cleanup-29722320-rlln9" Jul 06 12:00:00.365067 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:00.365009 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp67m\" (UniqueName: \"kubernetes.io/projected/bf2357b0-6395-4bb3-8bfd-6099d105a418-kube-api-access-dp67m\") pod \"maas-api-key-cleanup-29722320-rlln9\" (UID: \"bf2357b0-6395-4bb3-8bfd-6099d105a418\") " pod="opendatahub/maas-api-key-cleanup-29722320-rlln9" Jul 06 12:00:00.373807 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:00.373777 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp67m\" (UniqueName: \"kubernetes.io/projected/bf2357b0-6395-4bb3-8bfd-6099d105a418-kube-api-access-dp67m\") pod \"maas-api-key-cleanup-29722320-rlln9\" (UID: \"bf2357b0-6395-4bb3-8bfd-6099d105a418\") " pod="opendatahub/maas-api-key-cleanup-29722320-rlln9" Jul 06 12:00:00.459934 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:00.459836 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722320-rlln9" Jul 06 12:00:00.603253 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:00.603226 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722320-rlln9"] Jul 06 12:00:00.604980 ip-10-0-131-176 kubenswrapper[2568]: W0706 12:00:00.604954 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf2357b0_6395_4bb3_8bfd_6099d105a418.slice/crio-70547870cf88b28a0e2961bc8abe95f6b41a415bfefc980980c1aaf75e756d09 WatchSource:0}: Error finding container 70547870cf88b28a0e2961bc8abe95f6b41a415bfefc980980c1aaf75e756d09: Status 404 returned error can't find the container with id 70547870cf88b28a0e2961bc8abe95f6b41a415bfefc980980c1aaf75e756d09 Jul 06 12:00:01.600570 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:01.600536 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722320-rlln9" event={"ID":"bf2357b0-6395-4bb3-8bfd-6099d105a418","Type":"ContainerStarted","Data":"b68a3c798e11bd3a96627241745269fc9bd4c2da1308488282a64c526d5a3da8"} Jul 06 12:00:01.601001 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:01.600590 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722320-rlln9" event={"ID":"bf2357b0-6395-4bb3-8bfd-6099d105a418","Type":"ContainerStarted","Data":"70547870cf88b28a0e2961bc8abe95f6b41a415bfefc980980c1aaf75e756d09"} Jul 06 12:00:01.619440 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:01.619383 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29722320-rlln9" podStartSLOduration=1.619364883 podStartE2EDuration="1.619364883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:00:01.617202035 +0000 UTC m=+1900.097001051" watchObservedRunningTime="2026-07-06 12:00:01.619364883 +0000 UTC m=+1900.099163898" Jul 06 12:00:02.607931 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:02.607891 2568 generic.go:356] "Generic (PLEG): container finished" podID="bf2357b0-6395-4bb3-8bfd-6099d105a418" containerID="b68a3c798e11bd3a96627241745269fc9bd4c2da1308488282a64c526d5a3da8" exitCode=0 Jul 06 12:00:02.608367 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:02.607959 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722320-rlln9" event={"ID":"bf2357b0-6395-4bb3-8bfd-6099d105a418","Type":"ContainerDied","Data":"b68a3c798e11bd3a96627241745269fc9bd4c2da1308488282a64c526d5a3da8"} Jul 06 12:00:03.755555 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:03.755520 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722320-rlln9" Jul 06 12:00:03.797732 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:03.797694 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dp67m\" (UniqueName: \"kubernetes.io/projected/bf2357b0-6395-4bb3-8bfd-6099d105a418-kube-api-access-dp67m\") pod \"bf2357b0-6395-4bb3-8bfd-6099d105a418\" (UID: \"bf2357b0-6395-4bb3-8bfd-6099d105a418\") " Jul 06 12:00:03.799732 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:03.799708 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf2357b0-6395-4bb3-8bfd-6099d105a418-kube-api-access-dp67m" (OuterVolumeSpecName: "kube-api-access-dp67m") pod "bf2357b0-6395-4bb3-8bfd-6099d105a418" (UID: "bf2357b0-6395-4bb3-8bfd-6099d105a418"). InnerVolumeSpecName "kube-api-access-dp67m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:00:03.898989 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:03.898947 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp67m\" (UniqueName: \"kubernetes.io/projected/bf2357b0-6395-4bb3-8bfd-6099d105a418-kube-api-access-dp67m\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:00:04.620794 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:04.620764 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722320-rlln9" Jul 06 12:00:04.620794 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:04.620785 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722320-rlln9" event={"ID":"bf2357b0-6395-4bb3-8bfd-6099d105a418","Type":"ContainerDied","Data":"70547870cf88b28a0e2961bc8abe95f6b41a415bfefc980980c1aaf75e756d09"} Jul 06 12:00:04.621031 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:04.620831 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70547870cf88b28a0e2961bc8abe95f6b41a415bfefc980980c1aaf75e756d09" Jul 06 12:00:44.087094 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.087053 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw"] Jul 06 12:00:44.087549 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.087454 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf2357b0-6395-4bb3-8bfd-6099d105a418" containerName="cleanup" Jul 06 12:00:44.087549 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.087470 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf2357b0-6395-4bb3-8bfd-6099d105a418" containerName="cleanup" Jul 06 12:00:44.087549 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.087546 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf2357b0-6395-4bb3-8bfd-6099d105a418" containerName="cleanup" Jul 06 12:00:44.090790 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.090767 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:44.093795 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.093767 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6862d7-openshift-default-dockercfg-npm2h\"" Jul 06 12:00:44.102512 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.102480 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw"] Jul 06 12:00:44.154846 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.154813 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52bb6e49-bcfe-4403-9f0a-88b7fee49162-istio-envoy\") pod \"e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:44.155056 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.154858 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52bb6e49-bcfe-4403-9f0a-88b7fee49162-istio-data\") pod \"e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:44.155056 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.154890 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9trw\" (UniqueName: \"kubernetes.io/projected/52bb6e49-bcfe-4403-9f0a-88b7fee49162-kube-api-access-w9trw\") pod \"e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:44.155056 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.154990 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52bb6e49-bcfe-4403-9f0a-88b7fee49162-istio-podinfo\") pod \"e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:44.155056 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.155023 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bb6e49-bcfe-4403-9f0a-88b7fee49162-credential-socket\") pod \"e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:44.155056 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.155049 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52bb6e49-bcfe-4403-9f0a-88b7fee49162-istio-token\") pod \"e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:44.155287 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.155149 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bb6e49-bcfe-4403-9f0a-88b7fee49162-workload-certs\") pod \"e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:44.155287 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.155248 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bb6e49-bcfe-4403-9f0a-88b7fee49162-workload-socket\") pod \"e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:44.155287 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.155271 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52bb6e49-bcfe-4403-9f0a-88b7fee49162-istiod-ca-cert\") pod \"e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:44.256006 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.255964 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52bb6e49-bcfe-4403-9f0a-88b7fee49162-istio-envoy\") pod \"e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:44.256006 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.256005 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52bb6e49-bcfe-4403-9f0a-88b7fee49162-istio-data\") pod \"e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:44.256295 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.256027 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9trw\" (UniqueName: \"kubernetes.io/projected/52bb6e49-bcfe-4403-9f0a-88b7fee49162-kube-api-access-w9trw\") pod \"e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:44.256295 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.256049 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52bb6e49-bcfe-4403-9f0a-88b7fee49162-istio-podinfo\") pod \"e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:44.256295 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.256075 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bb6e49-bcfe-4403-9f0a-88b7fee49162-credential-socket\") pod \"e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:44.256295 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.256100 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52bb6e49-bcfe-4403-9f0a-88b7fee49162-istio-token\") pod \"e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:44.256295 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.256146 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bb6e49-bcfe-4403-9f0a-88b7fee49162-workload-certs\") pod \"e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:44.256295 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.256228 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bb6e49-bcfe-4403-9f0a-88b7fee49162-workload-socket\") pod \"e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:44.256295 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.256254 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52bb6e49-bcfe-4403-9f0a-88b7fee49162-istiod-ca-cert\") pod \"e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:44.256668 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.256473 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52bb6e49-bcfe-4403-9f0a-88b7fee49162-istio-data\") pod \"e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:44.256668 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.256557 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bb6e49-bcfe-4403-9f0a-88b7fee49162-credential-socket\") pod \"e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:44.256668 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.256642 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bb6e49-bcfe-4403-9f0a-88b7fee49162-workload-certs\") pod \"e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:44.256668 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.256648 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bb6e49-bcfe-4403-9f0a-88b7fee49162-workload-socket\") pod \"e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:44.257039 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.257016 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52bb6e49-bcfe-4403-9f0a-88b7fee49162-istiod-ca-cert\") pod \"e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:44.258469 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.258448 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52bb6e49-bcfe-4403-9f0a-88b7fee49162-istio-envoy\") pod \"e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:44.258770 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.258752 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52bb6e49-bcfe-4403-9f0a-88b7fee49162-istio-podinfo\") pod \"e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:44.266008 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.265985 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52bb6e49-bcfe-4403-9f0a-88b7fee49162-istio-token\") pod \"e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:44.266174 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.266151 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9trw\" (UniqueName: \"kubernetes.io/projected/52bb6e49-bcfe-4403-9f0a-88b7fee49162-kube-api-access-w9trw\") pod \"e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:44.401796 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.401760 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:44.540106 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.540072 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw"] Jul 06 12:00:44.541094 ip-10-0-131-176 kubenswrapper[2568]: W0706 12:00:44.541066 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52bb6e49_bcfe_4403_9f0a_88b7fee49162.slice/crio-b229631a6116425877fe13f3a1d390fce0368f72c043271dcda647ec833f0c94 WatchSource:0}: Error finding container b229631a6116425877fe13f3a1d390fce0368f72c043271dcda647ec833f0c94: Status 404 returned error can't find the container with id b229631a6116425877fe13f3a1d390fce0368f72c043271dcda647ec833f0c94 Jul 06 12:00:44.543379 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.543331 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 12:00:44.543489 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.543419 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 12:00:44.543489 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.543463 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 12:00:44.833808 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.833717 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" event={"ID":"52bb6e49-bcfe-4403-9f0a-88b7fee49162","Type":"ContainerStarted","Data":"345178f979abcc114e2918881f2cae4376515284db654d078479bdda179ad0a8"} Jul 06 12:00:44.833808 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.833772 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" event={"ID":"52bb6e49-bcfe-4403-9f0a-88b7fee49162","Type":"ContainerStarted","Data":"b229631a6116425877fe13f3a1d390fce0368f72c043271dcda647ec833f0c94"} Jul 06 12:00:44.855606 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:44.855540 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" podStartSLOduration=0.855520022 podStartE2EDuration="855.520022ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:00:44.855326428 +0000 UTC m=+1943.335125444" watchObservedRunningTime="2026-07-06 12:00:44.855520022 +0000 UTC m=+1943.335319040" Jul 06 12:00:45.402173 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:45.402139 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:45.407241 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:45.407213 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:45.840358 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:45.840262 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:45.841540 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:45.841517 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:00:46.950318 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:46.950286 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-6862d7-5555c7549-85kqg"] Jul 06 12:00:46.953901 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:46.953879 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-6862d7-5555c7549-85kqg" Jul 06 12:00:46.959342 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:46.959315 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-6862d7-serving-cert\"" Jul 06 12:00:46.981458 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:46.981418 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30af4427-2be2-46dc-9f7f-85c9cd1dd4b1-maas-api-tls\") pod \"maas-api-e2e-inf-a-6862d7-5555c7549-85kqg\" (UID: \"30af4427-2be2-46dc-9f7f-85c9cd1dd4b1\") " pod="opendatahub/maas-api-e2e-inf-a-6862d7-5555c7549-85kqg" Jul 06 12:00:46.981631 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:46.981469 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6n4g\" (UniqueName: \"kubernetes.io/projected/30af4427-2be2-46dc-9f7f-85c9cd1dd4b1-kube-api-access-j6n4g\") pod \"maas-api-e2e-inf-a-6862d7-5555c7549-85kqg\" (UID: \"30af4427-2be2-46dc-9f7f-85c9cd1dd4b1\") " pod="opendatahub/maas-api-e2e-inf-a-6862d7-5555c7549-85kqg" Jul 06 12:00:46.981631 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:46.981598 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30af4427-2be2-46dc-9f7f-85c9cd1dd4b1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6862d7-5555c7549-85kqg\" (UID: \"30af4427-2be2-46dc-9f7f-85c9cd1dd4b1\") " pod="opendatahub/maas-api-e2e-inf-a-6862d7-5555c7549-85kqg" Jul 06 12:00:46.983209 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:46.983189 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-6862d7-5555c7549-85kqg"] Jul 06 12:00:47.083105 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:47.083065 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30af4427-2be2-46dc-9f7f-85c9cd1dd4b1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6862d7-5555c7549-85kqg\" (UID: \"30af4427-2be2-46dc-9f7f-85c9cd1dd4b1\") " pod="opendatahub/maas-api-e2e-inf-a-6862d7-5555c7549-85kqg" Jul 06 12:00:47.083295 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:47.083123 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30af4427-2be2-46dc-9f7f-85c9cd1dd4b1-maas-api-tls\") pod \"maas-api-e2e-inf-a-6862d7-5555c7549-85kqg\" (UID: \"30af4427-2be2-46dc-9f7f-85c9cd1dd4b1\") " pod="opendatahub/maas-api-e2e-inf-a-6862d7-5555c7549-85kqg" Jul 06 12:00:47.083295 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:47.083146 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6n4g\" (UniqueName: \"kubernetes.io/projected/30af4427-2be2-46dc-9f7f-85c9cd1dd4b1-kube-api-access-j6n4g\") pod \"maas-api-e2e-inf-a-6862d7-5555c7549-85kqg\" (UID: \"30af4427-2be2-46dc-9f7f-85c9cd1dd4b1\") " pod="opendatahub/maas-api-e2e-inf-a-6862d7-5555c7549-85kqg" Jul 06 12:00:47.083790 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:47.083764 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30af4427-2be2-46dc-9f7f-85c9cd1dd4b1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6862d7-5555c7549-85kqg\" (UID: \"30af4427-2be2-46dc-9f7f-85c9cd1dd4b1\") " pod="opendatahub/maas-api-e2e-inf-a-6862d7-5555c7549-85kqg" Jul 06 12:00:47.085900 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:47.085874 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30af4427-2be2-46dc-9f7f-85c9cd1dd4b1-maas-api-tls\") pod \"maas-api-e2e-inf-a-6862d7-5555c7549-85kqg\" (UID: \"30af4427-2be2-46dc-9f7f-85c9cd1dd4b1\") " pod="opendatahub/maas-api-e2e-inf-a-6862d7-5555c7549-85kqg" Jul 06 12:00:47.095526 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:47.095482 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6n4g\" (UniqueName: \"kubernetes.io/projected/30af4427-2be2-46dc-9f7f-85c9cd1dd4b1-kube-api-access-j6n4g\") pod \"maas-api-e2e-inf-a-6862d7-5555c7549-85kqg\" (UID: \"30af4427-2be2-46dc-9f7f-85c9cd1dd4b1\") " pod="opendatahub/maas-api-e2e-inf-a-6862d7-5555c7549-85kqg" Jul 06 12:00:47.265111 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:47.265015 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-6862d7-5555c7549-85kqg" Jul 06 12:00:47.406570 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:47.406539 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-6862d7-5555c7549-85kqg"] Jul 06 12:00:47.407987 ip-10-0-131-176 kubenswrapper[2568]: W0706 12:00:47.407960 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30af4427_2be2_46dc_9f7f_85c9cd1dd4b1.slice/crio-87df3c482b409ec5c0317911665030690060535398b0c594051b36e9e10ab259 WatchSource:0}: Error finding container 87df3c482b409ec5c0317911665030690060535398b0c594051b36e9e10ab259: Status 404 returned error can't find the container with id 87df3c482b409ec5c0317911665030690060535398b0c594051b36e9e10ab259 Jul 06 12:00:47.409491 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:47.409472 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 12:00:47.853205 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:47.853168 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-6862d7-5555c7549-85kqg" event={"ID":"30af4427-2be2-46dc-9f7f-85c9cd1dd4b1","Type":"ContainerStarted","Data":"87df3c482b409ec5c0317911665030690060535398b0c594051b36e9e10ab259"} Jul 06 12:00:48.860742 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:48.860706 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-6862d7-5555c7549-85kqg" event={"ID":"30af4427-2be2-46dc-9f7f-85c9cd1dd4b1","Type":"ContainerStarted","Data":"3cc7ad66da078852aefccb0f1c41abd81f528a833d7510685ec3e25397be42e8"} Jul 06 12:00:48.861207 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:48.860769 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-6862d7-5555c7549-85kqg" Jul 06 12:00:48.878315 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:48.878246 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-6862d7-5555c7549-85kqg" podStartSLOduration=1.975821318 podStartE2EDuration="2.878226228s" totalImagesPullingTime="902.40491ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:00:48.877239334 +0000 UTC m=+1947.357038385" watchObservedRunningTime="2026-07-06 12:00:48.878226228 +0000 UTC m=+1947.358025244" Jul 06 12:00:54.871797 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:54.871765 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-6862d7-5555c7549-85kqg" Jul 06 12:00:55.353701 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:55.353651 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw"] Jul 06 12:00:55.361749 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:55.361725 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw" Jul 06 12:00:55.366182 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:55.366122 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-f369e4-serving-cert\"" Jul 06 12:00:55.366498 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:55.366472 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw"] Jul 06 12:00:55.461531 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:55.461497 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27abe9e6-d579-41e1-9e7f-807c0d1f5cdb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw\" (UID: \"27abe9e6-d579-41e1-9e7f-807c0d1f5cdb\") " pod="opendatahub/maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw" Jul 06 12:00:55.461972 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:55.461930 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwxbx\" (UniqueName: \"kubernetes.io/projected/27abe9e6-d579-41e1-9e7f-807c0d1f5cdb-kube-api-access-fwxbx\") pod \"maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw\" (UID: \"27abe9e6-d579-41e1-9e7f-807c0d1f5cdb\") " pod="opendatahub/maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw" Jul 06 12:00:55.462172 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:55.462148 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27abe9e6-d579-41e1-9e7f-807c0d1f5cdb-maas-api-tls\") pod \"maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw\" (UID: \"27abe9e6-d579-41e1-9e7f-807c0d1f5cdb\") " pod="opendatahub/maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw" Jul 06 12:00:55.563051 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:55.563012 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27abe9e6-d579-41e1-9e7f-807c0d1f5cdb-maas-api-tls\") pod \"maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw\" (UID: \"27abe9e6-d579-41e1-9e7f-807c0d1f5cdb\") " pod="opendatahub/maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw" Jul 06 12:00:55.563232 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:55.563125 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27abe9e6-d579-41e1-9e7f-807c0d1f5cdb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw\" (UID: \"27abe9e6-d579-41e1-9e7f-807c0d1f5cdb\") " pod="opendatahub/maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw" Jul 06 12:00:55.563232 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:55.563146 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwxbx\" (UniqueName: \"kubernetes.io/projected/27abe9e6-d579-41e1-9e7f-807c0d1f5cdb-kube-api-access-fwxbx\") pod \"maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw\" (UID: \"27abe9e6-d579-41e1-9e7f-807c0d1f5cdb\") " pod="opendatahub/maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw" Jul 06 12:00:55.563987 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:55.563964 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27abe9e6-d579-41e1-9e7f-807c0d1f5cdb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw\" (UID: \"27abe9e6-d579-41e1-9e7f-807c0d1f5cdb\") " pod="opendatahub/maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw" Jul 06 12:00:55.565514 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:55.565483 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27abe9e6-d579-41e1-9e7f-807c0d1f5cdb-maas-api-tls\") pod \"maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw\" (UID: \"27abe9e6-d579-41e1-9e7f-807c0d1f5cdb\") " pod="opendatahub/maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw" Jul 06 12:00:55.570930 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:55.570908 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwxbx\" (UniqueName: \"kubernetes.io/projected/27abe9e6-d579-41e1-9e7f-807c0d1f5cdb-kube-api-access-fwxbx\") pod \"maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw\" (UID: \"27abe9e6-d579-41e1-9e7f-807c0d1f5cdb\") " pod="opendatahub/maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw" Jul 06 12:00:55.676086 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:55.676050 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw" Jul 06 12:00:55.858394 ip-10-0-131-176 kubenswrapper[2568]: W0706 12:00:55.858333 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27abe9e6_d579_41e1_9e7f_807c0d1f5cdb.slice/crio-4e28b9b5959cd3921493cddb90b5cf9bf3518ba94879ab5b0e680a4d2257c2fe WatchSource:0}: Error finding container 4e28b9b5959cd3921493cddb90b5cf9bf3518ba94879ab5b0e680a4d2257c2fe: Status 404 returned error can't find the container with id 4e28b9b5959cd3921493cddb90b5cf9bf3518ba94879ab5b0e680a4d2257c2fe Jul 06 12:00:55.858567 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:55.858507 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw"] Jul 06 12:00:55.901825 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:55.901795 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw" event={"ID":"27abe9e6-d579-41e1-9e7f-807c0d1f5cdb","Type":"ContainerStarted","Data":"4e28b9b5959cd3921493cddb90b5cf9bf3518ba94879ab5b0e680a4d2257c2fe"} Jul 06 12:00:56.908668 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:56.908621 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw" event={"ID":"27abe9e6-d579-41e1-9e7f-807c0d1f5cdb","Type":"ContainerStarted","Data":"8381245ed29a4e51a89eea6abdcedc1b43bf22edf58e37f8853a58911d50a00b"} Jul 06 12:00:56.909089 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:56.908825 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw" Jul 06 12:00:56.926094 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:00:56.926047 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw" podStartSLOduration=1.5138594680000002 podStartE2EDuration="1.926033248s" totalImagesPullingTime="412.17378ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:00:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:00:56.923922158 +0000 UTC m=+1955.403721175" watchObservedRunningTime="2026-07-06 12:00:56.926033248 +0000 UTC m=+1955.405832263" Jul 06 12:01:00.995036 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:01:00.994123 2568 patch_prober.go:28] interesting pod/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": read tcp 10.132.0.2:55324->10.132.0.54:15021: read: connection reset by peer" start-of-body= Jul 06 12:01:00.995036 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:01:00.994181 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" podUID="52bb6e49-bcfe-4403-9f0a-88b7fee49162" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": read tcp 10.132.0.2:55324->10.132.0.54:15021: read: connection reset by peer" Jul 06 12:01:01.938980 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:01:01.938949 2568 generic.go:356] "Generic (PLEG): container finished" podID="52bb6e49-bcfe-4403-9f0a-88b7fee49162" containerID="345178f979abcc114e2918881f2cae4376515284db654d078479bdda179ad0a8" exitCode=0 Jul 06 12:01:01.938980 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:01:01.938989 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" event={"ID":"52bb6e49-bcfe-4403-9f0a-88b7fee49162","Type":"ContainerDied","Data":"345178f979abcc114e2918881f2cae4376515284db654d078479bdda179ad0a8"} Jul 06 12:01:01.939369 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:01:01.939354 2568 scope.go:117] "RemoveContainer" containerID="345178f979abcc114e2918881f2cae4376515284db654d078479bdda179ad0a8" Jul 06 12:01:01.940433 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:01:01.940403 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 12:01:01.940495 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:01:01.940472 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 12:01:01.940536 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:01:01.940503 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 12:01:02.919975 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:01:02.919944 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw" Jul 06 12:01:02.946853 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:01:02.946823 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" event={"ID":"52bb6e49-bcfe-4403-9f0a-88b7fee49162","Type":"ContainerStarted","Data":"41d557fc827cd2f73888856e3c02a20924d2e0dc379238ef604e9228709b1bf3"} Jul 06 12:01:03.402107 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:01:03.402063 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:01:03.402329 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:01:03.402310 2568 patch_prober.go:28] interesting pod/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 06 12:01:03.402381 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:01:03.402356 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" podUID="52bb6e49-bcfe-4403-9f0a-88b7fee49162" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 06 12:01:04.403153 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:01:04.403112 2568 patch_prober.go:28] interesting pod/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 06 12:01:04.403569 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:01:04.403175 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" podUID="52bb6e49-bcfe-4403-9f0a-88b7fee49162" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 06 12:01:05.402829 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:01:05.402793 2568 patch_prober.go:28] interesting pod/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 06 12:01:05.403007 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:01:05.402865 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" podUID="52bb6e49-bcfe-4403-9f0a-88b7fee49162" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 06 12:01:06.402395 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:01:06.402358 2568 patch_prober.go:28] interesting pod/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 06 12:01:06.402883 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:01:06.402429 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" podUID="52bb6e49-bcfe-4403-9f0a-88b7fee49162" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 06 12:01:07.405547 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:01:07.405517 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:01:07.405953 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:01:07.405882 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:01:07.406559 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:01:07.406539 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:02:20.584668 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:02:20.584626 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-6862d7-5555c7549-85kqg"] Jul 06 12:02:20.585353 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:02:20.584914 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-6862d7-5555c7549-85kqg" podUID="30af4427-2be2-46dc-9f7f-85c9cd1dd4b1" containerName="maas-api" containerID="cri-o://3cc7ad66da078852aefccb0f1c41abd81f528a833d7510685ec3e25397be42e8" gracePeriod=30 Jul 06 12:02:20.846667 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:02:20.846597 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-6862d7-5555c7549-85kqg" Jul 06 12:02:20.923031 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:02:20.922990 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30af4427-2be2-46dc-9f7f-85c9cd1dd4b1-openshift-service-ca\") pod \"30af4427-2be2-46dc-9f7f-85c9cd1dd4b1\" (UID: \"30af4427-2be2-46dc-9f7f-85c9cd1dd4b1\") " Jul 06 12:02:20.923216 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:02:20.923057 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30af4427-2be2-46dc-9f7f-85c9cd1dd4b1-maas-api-tls\") pod \"30af4427-2be2-46dc-9f7f-85c9cd1dd4b1\" (UID: \"30af4427-2be2-46dc-9f7f-85c9cd1dd4b1\") " Jul 06 12:02:20.923216 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:02:20.923156 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6n4g\" (UniqueName: \"kubernetes.io/projected/30af4427-2be2-46dc-9f7f-85c9cd1dd4b1-kube-api-access-j6n4g\") pod \"30af4427-2be2-46dc-9f7f-85c9cd1dd4b1\" (UID: \"30af4427-2be2-46dc-9f7f-85c9cd1dd4b1\") " Jul 06 12:02:20.923492 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:02:20.923458 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30af4427-2be2-46dc-9f7f-85c9cd1dd4b1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "30af4427-2be2-46dc-9f7f-85c9cd1dd4b1" (UID: "30af4427-2be2-46dc-9f7f-85c9cd1dd4b1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 12:02:20.925177 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:02:20.925154 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30af4427-2be2-46dc-9f7f-85c9cd1dd4b1-kube-api-access-j6n4g" (OuterVolumeSpecName: "kube-api-access-j6n4g") pod "30af4427-2be2-46dc-9f7f-85c9cd1dd4b1" (UID: "30af4427-2be2-46dc-9f7f-85c9cd1dd4b1"). InnerVolumeSpecName "kube-api-access-j6n4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:02:20.925258 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:02:20.925229 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30af4427-2be2-46dc-9f7f-85c9cd1dd4b1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "30af4427-2be2-46dc-9f7f-85c9cd1dd4b1" (UID: "30af4427-2be2-46dc-9f7f-85c9cd1dd4b1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:02:21.023858 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:02:21.023827 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6n4g\" (UniqueName: \"kubernetes.io/projected/30af4427-2be2-46dc-9f7f-85c9cd1dd4b1-kube-api-access-j6n4g\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:02:21.023858 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:02:21.023855 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30af4427-2be2-46dc-9f7f-85c9cd1dd4b1-openshift-service-ca\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:02:21.023858 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:02:21.023865 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30af4427-2be2-46dc-9f7f-85c9cd1dd4b1-maas-api-tls\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:02:21.396834 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:02:21.396799 2568 generic.go:356] "Generic (PLEG): container finished" podID="30af4427-2be2-46dc-9f7f-85c9cd1dd4b1" containerID="3cc7ad66da078852aefccb0f1c41abd81f528a833d7510685ec3e25397be42e8" exitCode=0 Jul 06 12:02:21.397024 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:02:21.396847 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-6862d7-5555c7549-85kqg" Jul 06 12:02:21.397024 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:02:21.396896 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-6862d7-5555c7549-85kqg" event={"ID":"30af4427-2be2-46dc-9f7f-85c9cd1dd4b1","Type":"ContainerDied","Data":"3cc7ad66da078852aefccb0f1c41abd81f528a833d7510685ec3e25397be42e8"} Jul 06 12:02:21.397024 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:02:21.396948 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-6862d7-5555c7549-85kqg" event={"ID":"30af4427-2be2-46dc-9f7f-85c9cd1dd4b1","Type":"ContainerDied","Data":"87df3c482b409ec5c0317911665030690060535398b0c594051b36e9e10ab259"} Jul 06 12:02:21.397024 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:02:21.396970 2568 scope.go:117] "RemoveContainer" containerID="3cc7ad66da078852aefccb0f1c41abd81f528a833d7510685ec3e25397be42e8" Jul 06 12:02:21.415241 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:02:21.415220 2568 scope.go:117] "RemoveContainer" containerID="3cc7ad66da078852aefccb0f1c41abd81f528a833d7510685ec3e25397be42e8" Jul 06 12:02:21.415536 ip-10-0-131-176 kubenswrapper[2568]: E0706 12:02:21.415514 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cc7ad66da078852aefccb0f1c41abd81f528a833d7510685ec3e25397be42e8\": container with ID starting with 3cc7ad66da078852aefccb0f1c41abd81f528a833d7510685ec3e25397be42e8 not found: ID does not exist" containerID="3cc7ad66da078852aefccb0f1c41abd81f528a833d7510685ec3e25397be42e8" Jul 06 12:02:21.415616 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:02:21.415544 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cc7ad66da078852aefccb0f1c41abd81f528a833d7510685ec3e25397be42e8"} err="failed to get container status \"3cc7ad66da078852aefccb0f1c41abd81f528a833d7510685ec3e25397be42e8\": rpc error: code = NotFound desc = could not find container \"3cc7ad66da078852aefccb0f1c41abd81f528a833d7510685ec3e25397be42e8\": container with ID starting with 3cc7ad66da078852aefccb0f1c41abd81f528a833d7510685ec3e25397be42e8 not found: ID does not exist" Jul 06 12:02:21.428145 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:02:21.428114 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-6862d7-5555c7549-85kqg"] Jul 06 12:02:21.432109 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:02:21.432087 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-6862d7-5555c7549-85kqg"] Jul 06 12:02:22.053166 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:02:22.053134 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30af4427-2be2-46dc-9f7f-85c9cd1dd4b1" path="/var/lib/kubelet/pods/30af4427-2be2-46dc-9f7f-85c9cd1dd4b1/volumes" Jul 06 12:03:02.925737 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:02.925699 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw"] Jul 06 12:03:02.926390 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:02.925926 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" podUID="52bb6e49-bcfe-4403-9f0a-88b7fee49162" containerName="istio-proxy" containerID="cri-o://41d557fc827cd2f73888856e3c02a20924d2e0dc379238ef604e9228709b1bf3" gracePeriod=30 Jul 06 12:03:04.309134 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:04.309097 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw"] Jul 06 12:03:04.309557 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:04.309311 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw" podUID="27abe9e6-d579-41e1-9e7f-807c0d1f5cdb" containerName="maas-api" containerID="cri-o://8381245ed29a4e51a89eea6abdcedc1b43bf22edf58e37f8853a58911d50a00b" gracePeriod=30 Jul 06 12:03:04.570327 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:04.570253 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw" Jul 06 12:03:04.620457 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:04.620417 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwxbx\" (UniqueName: \"kubernetes.io/projected/27abe9e6-d579-41e1-9e7f-807c0d1f5cdb-kube-api-access-fwxbx\") pod \"27abe9e6-d579-41e1-9e7f-807c0d1f5cdb\" (UID: \"27abe9e6-d579-41e1-9e7f-807c0d1f5cdb\") " Jul 06 12:03:04.620664 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:04.620470 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27abe9e6-d579-41e1-9e7f-807c0d1f5cdb-openshift-service-ca\") pod \"27abe9e6-d579-41e1-9e7f-807c0d1f5cdb\" (UID: \"27abe9e6-d579-41e1-9e7f-807c0d1f5cdb\") " Jul 06 12:03:04.620664 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:04.620524 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27abe9e6-d579-41e1-9e7f-807c0d1f5cdb-maas-api-tls\") pod \"27abe9e6-d579-41e1-9e7f-807c0d1f5cdb\" (UID: \"27abe9e6-d579-41e1-9e7f-807c0d1f5cdb\") " Jul 06 12:03:04.620915 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:04.620891 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27abe9e6-d579-41e1-9e7f-807c0d1f5cdb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "27abe9e6-d579-41e1-9e7f-807c0d1f5cdb" (UID: "27abe9e6-d579-41e1-9e7f-807c0d1f5cdb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 12:03:04.622735 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:04.622702 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27abe9e6-d579-41e1-9e7f-807c0d1f5cdb-kube-api-access-fwxbx" (OuterVolumeSpecName: "kube-api-access-fwxbx") pod "27abe9e6-d579-41e1-9e7f-807c0d1f5cdb" (UID: "27abe9e6-d579-41e1-9e7f-807c0d1f5cdb"). InnerVolumeSpecName "kube-api-access-fwxbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:03:04.622735 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:04.622723 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27abe9e6-d579-41e1-9e7f-807c0d1f5cdb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "27abe9e6-d579-41e1-9e7f-807c0d1f5cdb" (UID: "27abe9e6-d579-41e1-9e7f-807c0d1f5cdb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:03:04.631609 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:04.631570 2568 generic.go:356] "Generic (PLEG): container finished" podID="27abe9e6-d579-41e1-9e7f-807c0d1f5cdb" containerID="8381245ed29a4e51a89eea6abdcedc1b43bf22edf58e37f8853a58911d50a00b" exitCode=0 Jul 06 12:03:04.631765 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:04.631740 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw" event={"ID":"27abe9e6-d579-41e1-9e7f-807c0d1f5cdb","Type":"ContainerDied","Data":"8381245ed29a4e51a89eea6abdcedc1b43bf22edf58e37f8853a58911d50a00b"} Jul 06 12:03:04.631880 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:04.631808 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw" Jul 06 12:03:04.631880 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:04.631850 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw" event={"ID":"27abe9e6-d579-41e1-9e7f-807c0d1f5cdb","Type":"ContainerDied","Data":"4e28b9b5959cd3921493cddb90b5cf9bf3518ba94879ab5b0e680a4d2257c2fe"} Jul 06 12:03:04.631992 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:04.631903 2568 scope.go:117] "RemoveContainer" containerID="8381245ed29a4e51a89eea6abdcedc1b43bf22edf58e37f8853a58911d50a00b" Jul 06 12:03:04.641974 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:04.641866 2568 scope.go:117] "RemoveContainer" containerID="8381245ed29a4e51a89eea6abdcedc1b43bf22edf58e37f8853a58911d50a00b" Jul 06 12:03:04.642250 ip-10-0-131-176 kubenswrapper[2568]: E0706 12:03:04.642230 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8381245ed29a4e51a89eea6abdcedc1b43bf22edf58e37f8853a58911d50a00b\": container with ID starting with 8381245ed29a4e51a89eea6abdcedc1b43bf22edf58e37f8853a58911d50a00b not found: ID does not exist" containerID="8381245ed29a4e51a89eea6abdcedc1b43bf22edf58e37f8853a58911d50a00b" Jul 06 12:03:04.642330 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:04.642259 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8381245ed29a4e51a89eea6abdcedc1b43bf22edf58e37f8853a58911d50a00b"} err="failed to get container status \"8381245ed29a4e51a89eea6abdcedc1b43bf22edf58e37f8853a58911d50a00b\": rpc error: code = NotFound desc = could not find container \"8381245ed29a4e51a89eea6abdcedc1b43bf22edf58e37f8853a58911d50a00b\": container with ID starting with 8381245ed29a4e51a89eea6abdcedc1b43bf22edf58e37f8853a58911d50a00b not found: ID does not exist" Jul 06 12:03:04.660547 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:04.660514 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw"] Jul 06 12:03:04.667080 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:04.667041 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-f369e4-7cdc78bbc6-kbkmw"] Jul 06 12:03:04.721360 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:04.721314 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwxbx\" (UniqueName: \"kubernetes.io/projected/27abe9e6-d579-41e1-9e7f-807c0d1f5cdb-kube-api-access-fwxbx\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:03:04.721360 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:04.721352 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27abe9e6-d579-41e1-9e7f-807c0d1f5cdb-openshift-service-ca\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:03:04.721360 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:04.721364 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27abe9e6-d579-41e1-9e7f-807c0d1f5cdb-maas-api-tls\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:03:06.052735 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:06.052700 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27abe9e6-d579-41e1-9e7f-807c0d1f5cdb" path="/var/lib/kubelet/pods/27abe9e6-d579-41e1-9e7f-807c0d1f5cdb/volumes" Jul 06 12:03:07.408968 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:07.408933 2568 patch_prober.go:28] interesting pod/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 12:03:07.409361 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:07.409002 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" podUID="52bb6e49-bcfe-4403-9f0a-88b7fee49162" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 12:03:08.186886 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.186821 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:03:08.246922 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.246884 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52bb6e49-bcfe-4403-9f0a-88b7fee49162-istio-podinfo\") pod \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " Jul 06 12:03:08.246922 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.246924 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bb6e49-bcfe-4403-9f0a-88b7fee49162-credential-socket\") pod \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " Jul 06 12:03:08.247184 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.246948 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bb6e49-bcfe-4403-9f0a-88b7fee49162-workload-socket\") pod \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " Jul 06 12:03:08.247184 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.246992 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9trw\" (UniqueName: \"kubernetes.io/projected/52bb6e49-bcfe-4403-9f0a-88b7fee49162-kube-api-access-w9trw\") pod \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " Jul 06 12:03:08.247184 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.247029 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52bb6e49-bcfe-4403-9f0a-88b7fee49162-istio-data\") pod \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " Jul 06 12:03:08.247184 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.247049 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52bb6e49-bcfe-4403-9f0a-88b7fee49162-istio-token\") pod \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " Jul 06 12:03:08.247184 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.247082 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52bb6e49-bcfe-4403-9f0a-88b7fee49162-istio-envoy\") pod \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " Jul 06 12:03:08.247184 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.247135 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bb6e49-bcfe-4403-9f0a-88b7fee49162-workload-certs\") pod \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " Jul 06 12:03:08.247184 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.247164 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52bb6e49-bcfe-4403-9f0a-88b7fee49162-istiod-ca-cert\") pod \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\" (UID: \"52bb6e49-bcfe-4403-9f0a-88b7fee49162\") " Jul 06 12:03:08.247562 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.247229 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bb6e49-bcfe-4403-9f0a-88b7fee49162-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "52bb6e49-bcfe-4403-9f0a-88b7fee49162" (UID: "52bb6e49-bcfe-4403-9f0a-88b7fee49162"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:03:08.247562 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.247299 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bb6e49-bcfe-4403-9f0a-88b7fee49162-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "52bb6e49-bcfe-4403-9f0a-88b7fee49162" (UID: "52bb6e49-bcfe-4403-9f0a-88b7fee49162"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:03:08.247562 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.247453 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bb6e49-bcfe-4403-9f0a-88b7fee49162-credential-socket\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:03:08.247562 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.247471 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bb6e49-bcfe-4403-9f0a-88b7fee49162-workload-socket\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:03:08.247562 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.247500 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bb6e49-bcfe-4403-9f0a-88b7fee49162-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "52bb6e49-bcfe-4403-9f0a-88b7fee49162" (UID: "52bb6e49-bcfe-4403-9f0a-88b7fee49162"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:03:08.247562 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.247540 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52bb6e49-bcfe-4403-9f0a-88b7fee49162-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "52bb6e49-bcfe-4403-9f0a-88b7fee49162" (UID: "52bb6e49-bcfe-4403-9f0a-88b7fee49162"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 12:03:08.247957 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.247936 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bb6e49-bcfe-4403-9f0a-88b7fee49162-istio-data" (OuterVolumeSpecName: "istio-data") pod "52bb6e49-bcfe-4403-9f0a-88b7fee49162" (UID: "52bb6e49-bcfe-4403-9f0a-88b7fee49162"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:03:08.249412 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.249386 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52bb6e49-bcfe-4403-9f0a-88b7fee49162-istio-token" (OuterVolumeSpecName: "istio-token") pod "52bb6e49-bcfe-4403-9f0a-88b7fee49162" (UID: "52bb6e49-bcfe-4403-9f0a-88b7fee49162"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:03:08.249541 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.249487 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bb6e49-bcfe-4403-9f0a-88b7fee49162-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "52bb6e49-bcfe-4403-9f0a-88b7fee49162" (UID: "52bb6e49-bcfe-4403-9f0a-88b7fee49162"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:03:08.249647 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.249630 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52bb6e49-bcfe-4403-9f0a-88b7fee49162-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "52bb6e49-bcfe-4403-9f0a-88b7fee49162" (UID: "52bb6e49-bcfe-4403-9f0a-88b7fee49162"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 12:03:08.249720 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.249670 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52bb6e49-bcfe-4403-9f0a-88b7fee49162-kube-api-access-w9trw" (OuterVolumeSpecName: "kube-api-access-w9trw") pod "52bb6e49-bcfe-4403-9f0a-88b7fee49162" (UID: "52bb6e49-bcfe-4403-9f0a-88b7fee49162"). InnerVolumeSpecName "kube-api-access-w9trw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:03:08.347974 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.347931 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52bb6e49-bcfe-4403-9f0a-88b7fee49162-istio-data\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:03:08.347974 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.347964 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52bb6e49-bcfe-4403-9f0a-88b7fee49162-istio-token\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:03:08.347974 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.347973 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52bb6e49-bcfe-4403-9f0a-88b7fee49162-istio-envoy\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:03:08.347974 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.347983 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bb6e49-bcfe-4403-9f0a-88b7fee49162-workload-certs\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:03:08.348274 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.347992 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52bb6e49-bcfe-4403-9f0a-88b7fee49162-istiod-ca-cert\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:03:08.348274 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.348000 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52bb6e49-bcfe-4403-9f0a-88b7fee49162-istio-podinfo\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:03:08.348274 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.348008 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9trw\" (UniqueName: \"kubernetes.io/projected/52bb6e49-bcfe-4403-9f0a-88b7fee49162-kube-api-access-w9trw\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:03:08.656119 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.656076 2568 generic.go:356] "Generic (PLEG): container finished" podID="52bb6e49-bcfe-4403-9f0a-88b7fee49162" containerID="41d557fc827cd2f73888856e3c02a20924d2e0dc379238ef604e9228709b1bf3" exitCode=0 Jul 06 12:03:08.656591 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.656128 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" Jul 06 12:03:08.656591 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.656156 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" event={"ID":"52bb6e49-bcfe-4403-9f0a-88b7fee49162","Type":"ContainerDied","Data":"41d557fc827cd2f73888856e3c02a20924d2e0dc379238ef604e9228709b1bf3"} Jul 06 12:03:08.656591 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.656224 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw" event={"ID":"52bb6e49-bcfe-4403-9f0a-88b7fee49162","Type":"ContainerDied","Data":"b229631a6116425877fe13f3a1d390fce0368f72c043271dcda647ec833f0c94"} Jul 06 12:03:08.656591 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.656248 2568 scope.go:117] "RemoveContainer" containerID="41d557fc827cd2f73888856e3c02a20924d2e0dc379238ef604e9228709b1bf3" Jul 06 12:03:08.667166 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.667147 2568 scope.go:117] "RemoveContainer" containerID="345178f979abcc114e2918881f2cae4376515284db654d078479bdda179ad0a8" Jul 06 12:03:08.676265 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.676244 2568 scope.go:117] "RemoveContainer" containerID="41d557fc827cd2f73888856e3c02a20924d2e0dc379238ef604e9228709b1bf3" Jul 06 12:03:08.676558 ip-10-0-131-176 kubenswrapper[2568]: E0706 12:03:08.676536 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41d557fc827cd2f73888856e3c02a20924d2e0dc379238ef604e9228709b1bf3\": container with ID starting with 41d557fc827cd2f73888856e3c02a20924d2e0dc379238ef604e9228709b1bf3 not found: ID does not exist" containerID="41d557fc827cd2f73888856e3c02a20924d2e0dc379238ef604e9228709b1bf3" Jul 06 12:03:08.676651 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.676569 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d557fc827cd2f73888856e3c02a20924d2e0dc379238ef604e9228709b1bf3"} err="failed to get container status \"41d557fc827cd2f73888856e3c02a20924d2e0dc379238ef604e9228709b1bf3\": rpc error: code = NotFound desc = could not find container \"41d557fc827cd2f73888856e3c02a20924d2e0dc379238ef604e9228709b1bf3\": container with ID starting with 41d557fc827cd2f73888856e3c02a20924d2e0dc379238ef604e9228709b1bf3 not found: ID does not exist" Jul 06 12:03:08.676651 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.676591 2568 scope.go:117] "RemoveContainer" containerID="345178f979abcc114e2918881f2cae4376515284db654d078479bdda179ad0a8" Jul 06 12:03:08.676869 ip-10-0-131-176 kubenswrapper[2568]: E0706 12:03:08.676851 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"345178f979abcc114e2918881f2cae4376515284db654d078479bdda179ad0a8\": container with ID starting with 345178f979abcc114e2918881f2cae4376515284db654d078479bdda179ad0a8 not found: ID does not exist" containerID="345178f979abcc114e2918881f2cae4376515284db654d078479bdda179ad0a8" Jul 06 12:03:08.676923 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.676876 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"345178f979abcc114e2918881f2cae4376515284db654d078479bdda179ad0a8"} err="failed to get container status \"345178f979abcc114e2918881f2cae4376515284db654d078479bdda179ad0a8\": rpc error: code = NotFound desc = could not find container \"345178f979abcc114e2918881f2cae4376515284db654d078479bdda179ad0a8\": container with ID starting with 345178f979abcc114e2918881f2cae4376515284db654d078479bdda179ad0a8 not found: ID does not exist" Jul 06 12:03:08.694779 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.694758 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw"] Jul 06 12:03:08.705714 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:08.705688 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6862d7-openshift-default-66c8c884ff-cskhw"] Jul 06 12:03:10.052205 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:10.052172 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52bb6e49-bcfe-4403-9f0a-88b7fee49162" path="/var/lib/kubelet/pods/52bb6e49-bcfe-4403-9f0a-88b7fee49162/volumes" Jul 06 12:03:11.316709 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.316648 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25"] Jul 06 12:03:11.317228 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.317207 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30af4427-2be2-46dc-9f7f-85c9cd1dd4b1" containerName="maas-api" Jul 06 12:03:11.317315 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.317233 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="30af4427-2be2-46dc-9f7f-85c9cd1dd4b1" containerName="maas-api" Jul 06 12:03:11.317315 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.317251 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52bb6e49-bcfe-4403-9f0a-88b7fee49162" containerName="istio-proxy" Jul 06 12:03:11.317315 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.317260 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="52bb6e49-bcfe-4403-9f0a-88b7fee49162" containerName="istio-proxy" Jul 06 12:03:11.317315 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.317287 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27abe9e6-d579-41e1-9e7f-807c0d1f5cdb" containerName="maas-api" Jul 06 12:03:11.317315 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.317296 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="27abe9e6-d579-41e1-9e7f-807c0d1f5cdb" containerName="maas-api" Jul 06 12:03:11.317552 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.317317 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52bb6e49-bcfe-4403-9f0a-88b7fee49162" containerName="istio-proxy" Jul 06 12:03:11.317552 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.317326 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="52bb6e49-bcfe-4403-9f0a-88b7fee49162" containerName="istio-proxy" Jul 06 12:03:11.317552 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.317421 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="27abe9e6-d579-41e1-9e7f-807c0d1f5cdb" containerName="maas-api" Jul 06 12:03:11.317552 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.317438 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="30af4427-2be2-46dc-9f7f-85c9cd1dd4b1" containerName="maas-api" Jul 06 12:03:11.317552 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.317449 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="52bb6e49-bcfe-4403-9f0a-88b7fee49162" containerName="istio-proxy" Jul 06 12:03:11.317552 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.317459 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="52bb6e49-bcfe-4403-9f0a-88b7fee49162" containerName="istio-proxy" Jul 06 12:03:11.322033 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.322012 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:11.325042 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.325017 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ac7ca3-openshift-default-dockercfg-6bmm6\"" Jul 06 12:03:11.335633 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.335605 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25"] Jul 06 12:03:11.372917 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.372882 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d30c8c74-0270-4ec2-93fa-207cf13c42bb-istio-data\") pod \"e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:11.372917 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.372927 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d30c8c74-0270-4ec2-93fa-207cf13c42bb-workload-certs\") pod \"e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:11.373218 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.372966 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30c8c74-0270-4ec2-93fa-207cf13c42bb-workload-socket\") pod \"e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:11.373218 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.373007 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d30c8c74-0270-4ec2-93fa-207cf13c42bb-istiod-ca-cert\") pod \"e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:11.373218 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.373030 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzvng\" (UniqueName: \"kubernetes.io/projected/d30c8c74-0270-4ec2-93fa-207cf13c42bb-kube-api-access-xzvng\") pod \"e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:11.373218 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.373110 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d30c8c74-0270-4ec2-93fa-207cf13c42bb-istio-envoy\") pod \"e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:11.373218 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.373209 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30c8c74-0270-4ec2-93fa-207cf13c42bb-credential-socket\") pod \"e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:11.373444 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.373252 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d30c8c74-0270-4ec2-93fa-207cf13c42bb-istio-podinfo\") pod \"e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:11.373444 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.373284 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d30c8c74-0270-4ec2-93fa-207cf13c42bb-istio-token\") pod \"e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:11.474544 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.474500 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30c8c74-0270-4ec2-93fa-207cf13c42bb-credential-socket\") pod \"e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:11.474544 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.474551 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d30c8c74-0270-4ec2-93fa-207cf13c42bb-istio-podinfo\") pod \"e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:11.474880 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.474571 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d30c8c74-0270-4ec2-93fa-207cf13c42bb-istio-token\") pod \"e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:11.474880 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.474608 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d30c8c74-0270-4ec2-93fa-207cf13c42bb-istio-data\") pod \"e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:11.474880 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.474628 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d30c8c74-0270-4ec2-93fa-207cf13c42bb-workload-certs\") pod \"e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:11.474880 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.474649 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30c8c74-0270-4ec2-93fa-207cf13c42bb-workload-socket\") pod \"e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:11.474880 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.474668 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d30c8c74-0270-4ec2-93fa-207cf13c42bb-istiod-ca-cert\") pod \"e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:11.474880 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.474696 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzvng\" (UniqueName: \"kubernetes.io/projected/d30c8c74-0270-4ec2-93fa-207cf13c42bb-kube-api-access-xzvng\") pod \"e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:11.474880 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.474727 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d30c8c74-0270-4ec2-93fa-207cf13c42bb-istio-envoy\") pod \"e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:11.475231 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.475028 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30c8c74-0270-4ec2-93fa-207cf13c42bb-credential-socket\") pod \"e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:11.475231 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.475118 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d30c8c74-0270-4ec2-93fa-207cf13c42bb-workload-certs\") pod \"e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:11.475231 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.475157 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30c8c74-0270-4ec2-93fa-207cf13c42bb-workload-socket\") pod \"e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:11.475338 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.475290 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d30c8c74-0270-4ec2-93fa-207cf13c42bb-istio-data\") pod \"e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:11.475521 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.475496 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d30c8c74-0270-4ec2-93fa-207cf13c42bb-istiod-ca-cert\") pod \"e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:11.476974 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.476949 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d30c8c74-0270-4ec2-93fa-207cf13c42bb-istio-envoy\") pod \"e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:11.477486 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.477466 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d30c8c74-0270-4ec2-93fa-207cf13c42bb-istio-podinfo\") pod \"e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:11.483308 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.483286 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d30c8c74-0270-4ec2-93fa-207cf13c42bb-istio-token\") pod \"e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:11.483480 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.483459 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzvng\" (UniqueName: \"kubernetes.io/projected/d30c8c74-0270-4ec2-93fa-207cf13c42bb-kube-api-access-xzvng\") pod \"e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:11.634195 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.634092 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:11.781482 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.781456 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25"] Jul 06 12:03:11.783295 ip-10-0-131-176 kubenswrapper[2568]: W0706 12:03:11.783256 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd30c8c74_0270_4ec2_93fa_207cf13c42bb.slice/crio-7dcb4f8708324b0c1d408d814c9e53f4740b112357f4378e27232c8ff0ef508f WatchSource:0}: Error finding container 7dcb4f8708324b0c1d408d814c9e53f4740b112357f4378e27232c8ff0ef508f: Status 404 returned error can't find the container with id 7dcb4f8708324b0c1d408d814c9e53f4740b112357f4378e27232c8ff0ef508f Jul 06 12:03:11.785495 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.785453 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 12:03:11.785582 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.785553 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 12:03:11.785637 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:11.785598 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 12:03:12.680821 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:12.680788 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" event={"ID":"d30c8c74-0270-4ec2-93fa-207cf13c42bb","Type":"ContainerStarted","Data":"f6e48eee2c0d796ad4c0a8baf8504309708422ec06d6bba8783a0709eeed776b"} Jul 06 12:03:12.681291 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:12.680840 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" event={"ID":"d30c8c74-0270-4ec2-93fa-207cf13c42bb","Type":"ContainerStarted","Data":"7dcb4f8708324b0c1d408d814c9e53f4740b112357f4378e27232c8ff0ef508f"} Jul 06 12:03:12.706187 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:12.706139 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" podStartSLOduration=1.706122546 podStartE2EDuration="1.706122546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:03:12.704945833 +0000 UTC m=+2091.184744861" watchObservedRunningTime="2026-07-06 12:03:12.706122546 +0000 UTC m=+2091.185921562" Jul 06 12:03:13.634930 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:13.634894 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:13.640844 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:13.640815 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:13.687173 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:13.687132 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:13.688332 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:13.688307 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:14.839513 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:14.839477 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh"] Jul 06 12:03:14.843273 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:14.843255 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh" Jul 06 12:03:14.845975 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:14.845951 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-ac7ca3-serving-cert\"" Jul 06 12:03:14.850363 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:14.850336 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh"] Jul 06 12:03:14.907414 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:14.907379 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b-maas-api-tls\") pod \"maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh\" (UID: \"49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b\") " pod="opendatahub/maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh" Jul 06 12:03:14.907642 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:14.907422 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27jdt\" (UniqueName: \"kubernetes.io/projected/49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b-kube-api-access-27jdt\") pod \"maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh\" (UID: \"49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b\") " pod="opendatahub/maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh" Jul 06 12:03:14.907642 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:14.907548 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b-openshift-service-ca\") pod \"maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh\" (UID: \"49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b\") " pod="opendatahub/maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh" Jul 06 12:03:15.008815 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:15.008771 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b-maas-api-tls\") pod \"maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh\" (UID: \"49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b\") " pod="opendatahub/maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh" Jul 06 12:03:15.008815 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:15.008823 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27jdt\" (UniqueName: \"kubernetes.io/projected/49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b-kube-api-access-27jdt\") pod \"maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh\" (UID: \"49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b\") " pod="opendatahub/maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh" Jul 06 12:03:15.009081 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:15.008887 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b-openshift-service-ca\") pod \"maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh\" (UID: \"49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b\") " pod="opendatahub/maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh" Jul 06 12:03:15.009589 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:15.009566 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b-openshift-service-ca\") pod \"maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh\" (UID: \"49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b\") " pod="opendatahub/maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh" Jul 06 12:03:15.011955 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:15.011919 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b-maas-api-tls\") pod \"maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh\" (UID: \"49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b\") " pod="opendatahub/maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh" Jul 06 12:03:15.018061 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:15.018032 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27jdt\" (UniqueName: \"kubernetes.io/projected/49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b-kube-api-access-27jdt\") pod \"maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh\" (UID: \"49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b\") " pod="opendatahub/maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh" Jul 06 12:03:15.156735 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:15.156649 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh" Jul 06 12:03:15.305146 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:15.305121 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh"] Jul 06 12:03:15.307434 ip-10-0-131-176 kubenswrapper[2568]: W0706 12:03:15.307402 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49fbc84c_24d1_4e45_a1f7_9b2f0042ae7b.slice/crio-30cf3595693b6eab452029a1ac4f640787e9c9a09f62bf4b7564965a42818af9 WatchSource:0}: Error finding container 30cf3595693b6eab452029a1ac4f640787e9c9a09f62bf4b7564965a42818af9: Status 404 returned error can't find the container with id 30cf3595693b6eab452029a1ac4f640787e9c9a09f62bf4b7564965a42818af9 Jul 06 12:03:15.702755 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:15.702716 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh" event={"ID":"49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b","Type":"ContainerStarted","Data":"30cf3595693b6eab452029a1ac4f640787e9c9a09f62bf4b7564965a42818af9"} Jul 06 12:03:16.709594 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:16.709554 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh" event={"ID":"49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b","Type":"ContainerStarted","Data":"3b8670655127a599ed4341f778d03eccdea149a6ae0861c69bdcb9e2c7fb8898"} Jul 06 12:03:16.710119 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:16.709622 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh" Jul 06 12:03:16.729847 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:16.729785 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh" podStartSLOduration=2.233370412 podStartE2EDuration="2.729760709s" totalImagesPullingTime="496.390297ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:03:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:03:16.727728536 +0000 UTC m=+2095.207527555" watchObservedRunningTime="2026-07-06 12:03:16.729760709 +0000 UTC m=+2095.209559727" Jul 06 12:03:22.720491 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:22.720458 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh" Jul 06 12:03:23.044666 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:23.044563 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5"] Jul 06 12:03:23.049555 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:23.049524 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5" Jul 06 12:03:23.052490 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:23.052456 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-f43438-serving-cert\"" Jul 06 12:03:23.059706 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:23.059667 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5"] Jul 06 12:03:23.085258 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:23.085221 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97225ff2-e091-497e-a714-6b99f98771ae-maas-api-tls\") pod \"maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5\" (UID: \"97225ff2-e091-497e-a714-6b99f98771ae\") " pod="opendatahub/maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5" Jul 06 12:03:23.085454 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:23.085306 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97225ff2-e091-497e-a714-6b99f98771ae-openshift-service-ca\") pod \"maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5\" (UID: \"97225ff2-e091-497e-a714-6b99f98771ae\") " pod="opendatahub/maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5" Jul 06 12:03:23.085454 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:23.085432 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsjjt\" (UniqueName: \"kubernetes.io/projected/97225ff2-e091-497e-a714-6b99f98771ae-kube-api-access-zsjjt\") pod \"maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5\" (UID: \"97225ff2-e091-497e-a714-6b99f98771ae\") " pod="opendatahub/maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5" Jul 06 12:03:23.186758 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:23.186724 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97225ff2-e091-497e-a714-6b99f98771ae-openshift-service-ca\") pod \"maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5\" (UID: \"97225ff2-e091-497e-a714-6b99f98771ae\") " pod="opendatahub/maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5" Jul 06 12:03:23.186924 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:23.186789 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsjjt\" (UniqueName: \"kubernetes.io/projected/97225ff2-e091-497e-a714-6b99f98771ae-kube-api-access-zsjjt\") pod \"maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5\" (UID: \"97225ff2-e091-497e-a714-6b99f98771ae\") " pod="opendatahub/maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5" Jul 06 12:03:23.186924 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:23.186838 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97225ff2-e091-497e-a714-6b99f98771ae-maas-api-tls\") pod \"maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5\" (UID: \"97225ff2-e091-497e-a714-6b99f98771ae\") " pod="opendatahub/maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5" Jul 06 12:03:23.187031 ip-10-0-131-176 kubenswrapper[2568]: E0706 12:03:23.186954 2568 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-f43438-serving-cert: secret "maas-api-e2e-api-b-f43438-serving-cert" not found Jul 06 12:03:23.187031 ip-10-0-131-176 kubenswrapper[2568]: E0706 12:03:23.187020 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97225ff2-e091-497e-a714-6b99f98771ae-maas-api-tls podName:97225ff2-e091-497e-a714-6b99f98771ae nodeName:}" failed. No retries permitted until 2026-07-06 12:03:23.687001891 +0000 UTC m=+2102.166800885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/97225ff2-e091-497e-a714-6b99f98771ae-maas-api-tls") pod "maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5" (UID: "97225ff2-e091-497e-a714-6b99f98771ae") : secret "maas-api-e2e-api-b-f43438-serving-cert" not found Jul 06 12:03:23.187387 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:23.187368 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97225ff2-e091-497e-a714-6b99f98771ae-openshift-service-ca\") pod \"maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5\" (UID: \"97225ff2-e091-497e-a714-6b99f98771ae\") " pod="opendatahub/maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5" Jul 06 12:03:23.197012 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:23.196971 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsjjt\" (UniqueName: \"kubernetes.io/projected/97225ff2-e091-497e-a714-6b99f98771ae-kube-api-access-zsjjt\") pod \"maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5\" (UID: \"97225ff2-e091-497e-a714-6b99f98771ae\") " pod="opendatahub/maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5" Jul 06 12:03:23.690763 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:23.690724 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97225ff2-e091-497e-a714-6b99f98771ae-maas-api-tls\") pod \"maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5\" (UID: \"97225ff2-e091-497e-a714-6b99f98771ae\") " pod="opendatahub/maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5" Jul 06 12:03:23.690958 ip-10-0-131-176 kubenswrapper[2568]: E0706 12:03:23.690876 2568 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-f43438-serving-cert: secret "maas-api-e2e-api-b-f43438-serving-cert" not found Jul 06 12:03:23.690958 ip-10-0-131-176 kubenswrapper[2568]: E0706 12:03:23.690943 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97225ff2-e091-497e-a714-6b99f98771ae-maas-api-tls podName:97225ff2-e091-497e-a714-6b99f98771ae nodeName:}" failed. No retries permitted until 2026-07-06 12:03:24.690927902 +0000 UTC m=+2103.170726897 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/97225ff2-e091-497e-a714-6b99f98771ae-maas-api-tls") pod "maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5" (UID: "97225ff2-e091-497e-a714-6b99f98771ae") : secret "maas-api-e2e-api-b-f43438-serving-cert" not found Jul 06 12:03:24.702229 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:24.702186 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97225ff2-e091-497e-a714-6b99f98771ae-maas-api-tls\") pod \"maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5\" (UID: \"97225ff2-e091-497e-a714-6b99f98771ae\") " pod="opendatahub/maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5" Jul 06 12:03:24.704856 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:24.704818 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97225ff2-e091-497e-a714-6b99f98771ae-maas-api-tls\") pod \"maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5\" (UID: \"97225ff2-e091-497e-a714-6b99f98771ae\") " pod="opendatahub/maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5" Jul 06 12:03:24.862713 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:24.862626 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5" Jul 06 12:03:25.051030 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:25.044425 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5"] Jul 06 12:03:25.061981 ip-10-0-131-176 kubenswrapper[2568]: W0706 12:03:25.061927 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97225ff2_e091_497e_a714_6b99f98771ae.slice/crio-13de4f0b85d2cf73a815044f7dd265808a02bac704bcad6ca15214ca292dbb4f WatchSource:0}: Error finding container 13de4f0b85d2cf73a815044f7dd265808a02bac704bcad6ca15214ca292dbb4f: Status 404 returned error can't find the container with id 13de4f0b85d2cf73a815044f7dd265808a02bac704bcad6ca15214ca292dbb4f Jul 06 12:03:25.770096 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:25.769999 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5" event={"ID":"97225ff2-e091-497e-a714-6b99f98771ae","Type":"ContainerStarted","Data":"07b04ece397d948854baf90f31473801f401513bca700b00a785c99b27bb2063"} Jul 06 12:03:25.770096 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:25.770065 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5" event={"ID":"97225ff2-e091-497e-a714-6b99f98771ae","Type":"ContainerStarted","Data":"13de4f0b85d2cf73a815044f7dd265808a02bac704bcad6ca15214ca292dbb4f"} Jul 06 12:03:25.770531 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:25.770106 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5" Jul 06 12:03:25.790346 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:25.790292 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5" podStartSLOduration=2.381153199 podStartE2EDuration="2.790275824s" totalImagesPullingTime="409.122625ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:03:25.786955215 +0000 UTC m=+2104.266754222" watchObservedRunningTime="2026-07-06 12:03:25.790275824 +0000 UTC m=+2104.270074840" Jul 06 12:03:29.728915 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:29.728863 2568 patch_prober.go:28] interesting pod/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 06 12:03:29.729354 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:29.728935 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" podUID="d30c8c74-0270-4ec2-93fa-207cf13c42bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 12:03:30.798394 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:30.798356 2568 generic.go:356] "Generic (PLEG): container finished" podID="d30c8c74-0270-4ec2-93fa-207cf13c42bb" containerID="f6e48eee2c0d796ad4c0a8baf8504309708422ec06d6bba8783a0709eeed776b" exitCode=0 Jul 06 12:03:30.798847 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:30.798427 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" event={"ID":"d30c8c74-0270-4ec2-93fa-207cf13c42bb","Type":"ContainerDied","Data":"f6e48eee2c0d796ad4c0a8baf8504309708422ec06d6bba8783a0709eeed776b"} Jul 06 12:03:30.798895 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:30.798877 2568 scope.go:117] "RemoveContainer" containerID="f6e48eee2c0d796ad4c0a8baf8504309708422ec06d6bba8783a0709eeed776b" Jul 06 12:03:30.799920 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:30.799892 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 12:03:30.799995 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:30.799962 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 12:03:30.800035 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:30.799994 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 12:03:31.781004 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:31.780973 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5" Jul 06 12:03:31.806766 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:31.806718 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" event={"ID":"d30c8c74-0270-4ec2-93fa-207cf13c42bb","Type":"ContainerStarted","Data":"e66753cb8d70f976a69f0989af878a9b81c846eca4176e198388d1f221979a6c"} Jul 06 12:03:32.634644 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:32.634609 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:32.635035 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:32.635013 2568 patch_prober.go:28] interesting pod/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 06 12:03:32.635202 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:32.635180 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" podUID="d30c8c74-0270-4ec2-93fa-207cf13c42bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 06 12:03:33.635528 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:33.635478 2568 patch_prober.go:28] interesting pod/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 06 12:03:33.636122 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:33.635555 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" podUID="d30c8c74-0270-4ec2-93fa-207cf13c42bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 06 12:03:34.635214 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:34.635166 2568 patch_prober.go:28] interesting pod/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 06 12:03:34.635555 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:34.635242 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" podUID="d30c8c74-0270-4ec2-93fa-207cf13c42bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 06 12:03:35.639312 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:35.639280 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:35.639747 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:35.639709 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:35.640434 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:35.640413 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:36.155508 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:36.155469 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh"] Jul 06 12:03:36.155770 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:36.155742 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh" podUID="49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b" containerName="maas-api" containerID="cri-o://3b8670655127a599ed4341f778d03eccdea149a6ae0861c69bdcb9e2c7fb8898" gracePeriod=30 Jul 06 12:03:36.441665 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:36.441638 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh" Jul 06 12:03:36.516171 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:36.516127 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27jdt\" (UniqueName: \"kubernetes.io/projected/49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b-kube-api-access-27jdt\") pod \"49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b\" (UID: \"49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b\") " Jul 06 12:03:36.516394 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:36.516229 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b-maas-api-tls\") pod \"49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b\" (UID: \"49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b\") " Jul 06 12:03:36.516394 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:36.516255 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b-openshift-service-ca\") pod \"49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b\" (UID: \"49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b\") " Jul 06 12:03:36.516736 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:36.516710 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b" (UID: "49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 12:03:36.518360 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:36.518328 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b" (UID: "49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:03:36.518461 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:36.518364 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b-kube-api-access-27jdt" (OuterVolumeSpecName: "kube-api-access-27jdt") pod "49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b" (UID: "49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b"). InnerVolumeSpecName "kube-api-access-27jdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:03:36.617329 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:36.617286 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27jdt\" (UniqueName: \"kubernetes.io/projected/49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b-kube-api-access-27jdt\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:03:36.617329 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:36.617323 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b-maas-api-tls\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:03:36.617329 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:36.617333 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b-openshift-service-ca\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:03:36.838430 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:36.838322 2568 generic.go:356] "Generic (PLEG): container finished" podID="49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b" containerID="3b8670655127a599ed4341f778d03eccdea149a6ae0861c69bdcb9e2c7fb8898" exitCode=0 Jul 06 12:03:36.838430 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:36.838366 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh" Jul 06 12:03:36.838430 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:36.838413 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh" event={"ID":"49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b","Type":"ContainerDied","Data":"3b8670655127a599ed4341f778d03eccdea149a6ae0861c69bdcb9e2c7fb8898"} Jul 06 12:03:36.839051 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:36.838492 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh" event={"ID":"49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b","Type":"ContainerDied","Data":"30cf3595693b6eab452029a1ac4f640787e9c9a09f62bf4b7564965a42818af9"} Jul 06 12:03:36.839051 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:36.838516 2568 scope.go:117] "RemoveContainer" containerID="3b8670655127a599ed4341f778d03eccdea149a6ae0861c69bdcb9e2c7fb8898" Jul 06 12:03:36.848452 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:36.848434 2568 scope.go:117] "RemoveContainer" containerID="3b8670655127a599ed4341f778d03eccdea149a6ae0861c69bdcb9e2c7fb8898" Jul 06 12:03:36.848737 ip-10-0-131-176 kubenswrapper[2568]: E0706 12:03:36.848718 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b8670655127a599ed4341f778d03eccdea149a6ae0861c69bdcb9e2c7fb8898\": container with ID starting with 3b8670655127a599ed4341f778d03eccdea149a6ae0861c69bdcb9e2c7fb8898 not found: ID does not exist" containerID="3b8670655127a599ed4341f778d03eccdea149a6ae0861c69bdcb9e2c7fb8898" Jul 06 12:03:36.848791 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:36.848745 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b8670655127a599ed4341f778d03eccdea149a6ae0861c69bdcb9e2c7fb8898"} err="failed to get container status \"3b8670655127a599ed4341f778d03eccdea149a6ae0861c69bdcb9e2c7fb8898\": rpc error: code = NotFound desc = could not find container \"3b8670655127a599ed4341f778d03eccdea149a6ae0861c69bdcb9e2c7fb8898\": container with ID starting with 3b8670655127a599ed4341f778d03eccdea149a6ae0861c69bdcb9e2c7fb8898 not found: ID does not exist" Jul 06 12:03:36.866022 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:36.865991 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh"] Jul 06 12:03:36.870108 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:36.870085 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-ac7ca3-84d78c87b9-sk6zh"] Jul 06 12:03:38.052244 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:38.052202 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b" path="/var/lib/kubelet/pods/49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b/volumes" Jul 06 12:03:42.064429 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:42.064394 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25"] Jul 06 12:03:42.064898 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:42.064588 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" podUID="d30c8c74-0270-4ec2-93fa-207cf13c42bb" containerName="istio-proxy" containerID="cri-o://e66753cb8d70f976a69f0989af878a9b81c846eca4176e198388d1f221979a6c" gracePeriod=30 Jul 06 12:03:43.206515 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:43.206460 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5"] Jul 06 12:03:43.207382 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:43.207321 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5" podUID="97225ff2-e091-497e-a714-6b99f98771ae" containerName="maas-api" containerID="cri-o://07b04ece397d948854baf90f31473801f401513bca700b00a785c99b27bb2063" gracePeriod=30 Jul 06 12:03:43.452687 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:43.452645 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5" Jul 06 12:03:43.578731 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:43.578594 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97225ff2-e091-497e-a714-6b99f98771ae-maas-api-tls\") pod \"97225ff2-e091-497e-a714-6b99f98771ae\" (UID: \"97225ff2-e091-497e-a714-6b99f98771ae\") " Jul 06 12:03:43.578731 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:43.578653 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97225ff2-e091-497e-a714-6b99f98771ae-openshift-service-ca\") pod \"97225ff2-e091-497e-a714-6b99f98771ae\" (UID: \"97225ff2-e091-497e-a714-6b99f98771ae\") " Jul 06 12:03:43.579002 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:43.578797 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsjjt\" (UniqueName: \"kubernetes.io/projected/97225ff2-e091-497e-a714-6b99f98771ae-kube-api-access-zsjjt\") pod \"97225ff2-e091-497e-a714-6b99f98771ae\" (UID: \"97225ff2-e091-497e-a714-6b99f98771ae\") " Jul 06 12:03:43.579154 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:43.579125 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97225ff2-e091-497e-a714-6b99f98771ae-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "97225ff2-e091-497e-a714-6b99f98771ae" (UID: "97225ff2-e091-497e-a714-6b99f98771ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 12:03:43.580896 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:43.580872 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97225ff2-e091-497e-a714-6b99f98771ae-kube-api-access-zsjjt" (OuterVolumeSpecName: "kube-api-access-zsjjt") pod "97225ff2-e091-497e-a714-6b99f98771ae" (UID: "97225ff2-e091-497e-a714-6b99f98771ae"). InnerVolumeSpecName "kube-api-access-zsjjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:03:43.580970 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:43.580919 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97225ff2-e091-497e-a714-6b99f98771ae-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "97225ff2-e091-497e-a714-6b99f98771ae" (UID: "97225ff2-e091-497e-a714-6b99f98771ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:03:43.680019 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:43.679974 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsjjt\" (UniqueName: \"kubernetes.io/projected/97225ff2-e091-497e-a714-6b99f98771ae-kube-api-access-zsjjt\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:03:43.680019 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:43.680013 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97225ff2-e091-497e-a714-6b99f98771ae-maas-api-tls\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:03:43.680019 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:43.680028 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97225ff2-e091-497e-a714-6b99f98771ae-openshift-service-ca\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:03:43.883238 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:43.883131 2568 generic.go:356] "Generic (PLEG): container finished" podID="97225ff2-e091-497e-a714-6b99f98771ae" containerID="07b04ece397d948854baf90f31473801f401513bca700b00a785c99b27bb2063" exitCode=0 Jul 06 12:03:43.883238 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:43.883183 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5" Jul 06 12:03:43.883238 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:43.883216 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5" event={"ID":"97225ff2-e091-497e-a714-6b99f98771ae","Type":"ContainerDied","Data":"07b04ece397d948854baf90f31473801f401513bca700b00a785c99b27bb2063"} Jul 06 12:03:43.883491 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:43.883276 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5" event={"ID":"97225ff2-e091-497e-a714-6b99f98771ae","Type":"ContainerDied","Data":"13de4f0b85d2cf73a815044f7dd265808a02bac704bcad6ca15214ca292dbb4f"} Jul 06 12:03:43.883491 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:43.883304 2568 scope.go:117] "RemoveContainer" containerID="07b04ece397d948854baf90f31473801f401513bca700b00a785c99b27bb2063" Jul 06 12:03:43.893119 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:43.893097 2568 scope.go:117] "RemoveContainer" containerID="07b04ece397d948854baf90f31473801f401513bca700b00a785c99b27bb2063" Jul 06 12:03:43.893447 ip-10-0-131-176 kubenswrapper[2568]: E0706 12:03:43.893425 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07b04ece397d948854baf90f31473801f401513bca700b00a785c99b27bb2063\": container with ID starting with 07b04ece397d948854baf90f31473801f401513bca700b00a785c99b27bb2063 not found: ID does not exist" containerID="07b04ece397d948854baf90f31473801f401513bca700b00a785c99b27bb2063" Jul 06 12:03:43.893502 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:43.893456 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b04ece397d948854baf90f31473801f401513bca700b00a785c99b27bb2063"} err="failed to get container status \"07b04ece397d948854baf90f31473801f401513bca700b00a785c99b27bb2063\": rpc error: code = NotFound desc = could not find container \"07b04ece397d948854baf90f31473801f401513bca700b00a785c99b27bb2063\": container with ID starting with 07b04ece397d948854baf90f31473801f401513bca700b00a785c99b27bb2063 not found: ID does not exist" Jul 06 12:03:43.906491 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:43.906463 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5"] Jul 06 12:03:43.910861 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:43.910836 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-f43438-5bbcd4857b-8s4r5"] Jul 06 12:03:44.053092 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:44.053057 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97225ff2-e091-497e-a714-6b99f98771ae" path="/var/lib/kubelet/pods/97225ff2-e091-497e-a714-6b99f98771ae/volumes" Jul 06 12:03:47.322890 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.322862 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:47.414498 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.414457 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30c8c74-0270-4ec2-93fa-207cf13c42bb-credential-socket\") pod \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " Jul 06 12:03:47.414743 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.414517 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30c8c74-0270-4ec2-93fa-207cf13c42bb-workload-socket\") pod \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " Jul 06 12:03:47.414743 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.414543 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d30c8c74-0270-4ec2-93fa-207cf13c42bb-workload-certs\") pod \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " Jul 06 12:03:47.414743 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.414578 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d30c8c74-0270-4ec2-93fa-207cf13c42bb-istio-data\") pod \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " Jul 06 12:03:47.414743 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.414602 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d30c8c74-0270-4ec2-93fa-207cf13c42bb-istio-podinfo\") pod \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " Jul 06 12:03:47.414743 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.414672 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d30c8c74-0270-4ec2-93fa-207cf13c42bb-istiod-ca-cert\") pod \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " Jul 06 12:03:47.414743 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.414718 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzvng\" (UniqueName: \"kubernetes.io/projected/d30c8c74-0270-4ec2-93fa-207cf13c42bb-kube-api-access-xzvng\") pod \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " Jul 06 12:03:47.414743 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.414741 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d30c8c74-0270-4ec2-93fa-207cf13c42bb-istio-token\") pod \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " Jul 06 12:03:47.415125 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.414768 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d30c8c74-0270-4ec2-93fa-207cf13c42bb-istio-envoy\") pod \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\" (UID: \"d30c8c74-0270-4ec2-93fa-207cf13c42bb\") " Jul 06 12:03:47.415125 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.414812 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30c8c74-0270-4ec2-93fa-207cf13c42bb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d30c8c74-0270-4ec2-93fa-207cf13c42bb" (UID: "d30c8c74-0270-4ec2-93fa-207cf13c42bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:03:47.415125 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.414942 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30c8c74-0270-4ec2-93fa-207cf13c42bb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d30c8c74-0270-4ec2-93fa-207cf13c42bb" (UID: "d30c8c74-0270-4ec2-93fa-207cf13c42bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:03:47.415125 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.414962 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30c8c74-0270-4ec2-93fa-207cf13c42bb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d30c8c74-0270-4ec2-93fa-207cf13c42bb" (UID: "d30c8c74-0270-4ec2-93fa-207cf13c42bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:03:47.415125 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.415103 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30c8c74-0270-4ec2-93fa-207cf13c42bb-credential-socket\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:03:47.415125 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.415128 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30c8c74-0270-4ec2-93fa-207cf13c42bb-workload-socket\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:03:47.415458 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.415144 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d30c8c74-0270-4ec2-93fa-207cf13c42bb-workload-certs\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:03:47.415458 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.415308 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d30c8c74-0270-4ec2-93fa-207cf13c42bb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d30c8c74-0270-4ec2-93fa-207cf13c42bb" (UID: "d30c8c74-0270-4ec2-93fa-207cf13c42bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 12:03:47.415569 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.415506 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30c8c74-0270-4ec2-93fa-207cf13c42bb-istio-data" (OuterVolumeSpecName: "istio-data") pod "d30c8c74-0270-4ec2-93fa-207cf13c42bb" (UID: "d30c8c74-0270-4ec2-93fa-207cf13c42bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:03:47.417141 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.417112 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d30c8c74-0270-4ec2-93fa-207cf13c42bb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d30c8c74-0270-4ec2-93fa-207cf13c42bb" (UID: "d30c8c74-0270-4ec2-93fa-207cf13c42bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 12:03:47.417141 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.417129 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d30c8c74-0270-4ec2-93fa-207cf13c42bb-istio-token" (OuterVolumeSpecName: "istio-token") pod "d30c8c74-0270-4ec2-93fa-207cf13c42bb" (UID: "d30c8c74-0270-4ec2-93fa-207cf13c42bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:03:47.417299 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.417139 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30c8c74-0270-4ec2-93fa-207cf13c42bb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d30c8c74-0270-4ec2-93fa-207cf13c42bb" (UID: "d30c8c74-0270-4ec2-93fa-207cf13c42bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:03:47.417299 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.417149 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d30c8c74-0270-4ec2-93fa-207cf13c42bb-kube-api-access-xzvng" (OuterVolumeSpecName: "kube-api-access-xzvng") pod "d30c8c74-0270-4ec2-93fa-207cf13c42bb" (UID: "d30c8c74-0270-4ec2-93fa-207cf13c42bb"). InnerVolumeSpecName "kube-api-access-xzvng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:03:47.516032 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.515994 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d30c8c74-0270-4ec2-93fa-207cf13c42bb-istiod-ca-cert\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:03:47.516032 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.516025 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzvng\" (UniqueName: \"kubernetes.io/projected/d30c8c74-0270-4ec2-93fa-207cf13c42bb-kube-api-access-xzvng\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:03:47.516032 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.516036 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d30c8c74-0270-4ec2-93fa-207cf13c42bb-istio-token\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:03:47.516032 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.516044 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d30c8c74-0270-4ec2-93fa-207cf13c42bb-istio-envoy\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:03:47.516331 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.516052 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d30c8c74-0270-4ec2-93fa-207cf13c42bb-istio-data\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:03:47.516331 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.516060 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d30c8c74-0270-4ec2-93fa-207cf13c42bb-istio-podinfo\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:03:47.908374 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.908341 2568 generic.go:356] "Generic (PLEG): container finished" podID="d30c8c74-0270-4ec2-93fa-207cf13c42bb" containerID="e66753cb8d70f976a69f0989af878a9b81c846eca4176e198388d1f221979a6c" exitCode=0 Jul 06 12:03:47.908585 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.908392 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" Jul 06 12:03:47.908585 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.908425 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" event={"ID":"d30c8c74-0270-4ec2-93fa-207cf13c42bb","Type":"ContainerDied","Data":"e66753cb8d70f976a69f0989af878a9b81c846eca4176e198388d1f221979a6c"} Jul 06 12:03:47.908585 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.908524 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25" event={"ID":"d30c8c74-0270-4ec2-93fa-207cf13c42bb","Type":"ContainerDied","Data":"7dcb4f8708324b0c1d408d814c9e53f4740b112357f4378e27232c8ff0ef508f"} Jul 06 12:03:47.908585 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.908554 2568 scope.go:117] "RemoveContainer" containerID="e66753cb8d70f976a69f0989af878a9b81c846eca4176e198388d1f221979a6c" Jul 06 12:03:47.918219 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.918201 2568 scope.go:117] "RemoveContainer" containerID="f6e48eee2c0d796ad4c0a8baf8504309708422ec06d6bba8783a0709eeed776b" Jul 06 12:03:47.926441 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.926422 2568 scope.go:117] "RemoveContainer" containerID="e66753cb8d70f976a69f0989af878a9b81c846eca4176e198388d1f221979a6c" Jul 06 12:03:47.926697 ip-10-0-131-176 kubenswrapper[2568]: E0706 12:03:47.926661 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e66753cb8d70f976a69f0989af878a9b81c846eca4176e198388d1f221979a6c\": container with ID starting with e66753cb8d70f976a69f0989af878a9b81c846eca4176e198388d1f221979a6c not found: ID does not exist" containerID="e66753cb8d70f976a69f0989af878a9b81c846eca4176e198388d1f221979a6c" Jul 06 12:03:47.926745 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.926709 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e66753cb8d70f976a69f0989af878a9b81c846eca4176e198388d1f221979a6c"} err="failed to get container status \"e66753cb8d70f976a69f0989af878a9b81c846eca4176e198388d1f221979a6c\": rpc error: code = NotFound desc = could not find container \"e66753cb8d70f976a69f0989af878a9b81c846eca4176e198388d1f221979a6c\": container with ID starting with e66753cb8d70f976a69f0989af878a9b81c846eca4176e198388d1f221979a6c not found: ID does not exist" Jul 06 12:03:47.926745 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.926728 2568 scope.go:117] "RemoveContainer" containerID="f6e48eee2c0d796ad4c0a8baf8504309708422ec06d6bba8783a0709eeed776b" Jul 06 12:03:47.926989 ip-10-0-131-176 kubenswrapper[2568]: E0706 12:03:47.926968 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6e48eee2c0d796ad4c0a8baf8504309708422ec06d6bba8783a0709eeed776b\": container with ID starting with f6e48eee2c0d796ad4c0a8baf8504309708422ec06d6bba8783a0709eeed776b not found: ID does not exist" containerID="f6e48eee2c0d796ad4c0a8baf8504309708422ec06d6bba8783a0709eeed776b" Jul 06 12:03:47.927026 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.926996 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6e48eee2c0d796ad4c0a8baf8504309708422ec06d6bba8783a0709eeed776b"} err="failed to get container status \"f6e48eee2c0d796ad4c0a8baf8504309708422ec06d6bba8783a0709eeed776b\": rpc error: code = NotFound desc = could not find container \"f6e48eee2c0d796ad4c0a8baf8504309708422ec06d6bba8783a0709eeed776b\": container with ID starting with f6e48eee2c0d796ad4c0a8baf8504309708422ec06d6bba8783a0709eeed776b not found: ID does not exist" Jul 06 12:03:47.932278 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.932243 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25"] Jul 06 12:03:47.935942 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:47.935907 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ac7ca3-openshift-default-668479f687-c7k25"] Jul 06 12:03:48.053997 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:48.053963 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d30c8c74-0270-4ec2-93fa-207cf13c42bb" path="/var/lib/kubelet/pods/d30c8c74-0270-4ec2-93fa-207cf13c42bb/volumes" Jul 06 12:03:50.000896 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.000856 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc"] Jul 06 12:03:50.001435 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.001408 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97225ff2-e091-497e-a714-6b99f98771ae" containerName="maas-api" Jul 06 12:03:50.001435 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.001436 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="97225ff2-e091-497e-a714-6b99f98771ae" containerName="maas-api" Jul 06 12:03:50.001697 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.001455 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d30c8c74-0270-4ec2-93fa-207cf13c42bb" containerName="istio-proxy" Jul 06 12:03:50.001697 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.001461 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d30c8c74-0270-4ec2-93fa-207cf13c42bb" containerName="istio-proxy" Jul 06 12:03:50.001697 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.001471 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d30c8c74-0270-4ec2-93fa-207cf13c42bb" containerName="istio-proxy" Jul 06 12:03:50.001697 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.001476 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d30c8c74-0270-4ec2-93fa-207cf13c42bb" containerName="istio-proxy" Jul 06 12:03:50.001697 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.001486 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b" containerName="maas-api" Jul 06 12:03:50.001697 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.001493 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b" containerName="maas-api" Jul 06 12:03:50.001697 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.001596 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="49fbc84c-24d1-4e45-a1f7-9b2f0042ae7b" containerName="maas-api" Jul 06 12:03:50.001697 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.001606 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="97225ff2-e091-497e-a714-6b99f98771ae" containerName="maas-api" Jul 06 12:03:50.001697 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.001612 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d30c8c74-0270-4ec2-93fa-207cf13c42bb" containerName="istio-proxy" Jul 06 12:03:50.001697 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.001619 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d30c8c74-0270-4ec2-93fa-207cf13c42bb" containerName="istio-proxy" Jul 06 12:03:50.006217 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.006193 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:50.009022 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.009000 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-8b7fb6-openshift-default-dockercfg-nmn5h\"" Jul 06 12:03:50.017157 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.017129 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc"] Jul 06 12:03:50.140382 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.140347 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55051609-ecc0-4faf-86ac-d4b04ce9dce5-istio-podinfo\") pod \"e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:50.140578 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.140405 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55051609-ecc0-4faf-86ac-d4b04ce9dce5-workload-socket\") pod \"e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:50.140578 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.140436 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55051609-ecc0-4faf-86ac-d4b04ce9dce5-credential-socket\") pod \"e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:50.140578 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.140485 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55051609-ecc0-4faf-86ac-d4b04ce9dce5-istio-data\") pod \"e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:50.140578 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.140511 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55051609-ecc0-4faf-86ac-d4b04ce9dce5-istio-envoy\") pod \"e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:50.140578 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.140530 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55051609-ecc0-4faf-86ac-d4b04ce9dce5-istiod-ca-cert\") pod \"e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:50.140866 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.140601 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55051609-ecc0-4faf-86ac-d4b04ce9dce5-istio-token\") pod \"e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:50.140866 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.140638 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrdrs\" (UniqueName: \"kubernetes.io/projected/55051609-ecc0-4faf-86ac-d4b04ce9dce5-kube-api-access-xrdrs\") pod \"e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:50.140866 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.140661 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55051609-ecc0-4faf-86ac-d4b04ce9dce5-workload-certs\") pod \"e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:50.241278 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.241240 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55051609-ecc0-4faf-86ac-d4b04ce9dce5-istio-token\") pod \"e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:50.241278 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.241294 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrdrs\" (UniqueName: \"kubernetes.io/projected/55051609-ecc0-4faf-86ac-d4b04ce9dce5-kube-api-access-xrdrs\") pod \"e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:50.241559 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.241323 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55051609-ecc0-4faf-86ac-d4b04ce9dce5-workload-certs\") pod \"e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:50.241559 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.241382 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55051609-ecc0-4faf-86ac-d4b04ce9dce5-istio-podinfo\") pod \"e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:50.241559 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.241442 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55051609-ecc0-4faf-86ac-d4b04ce9dce5-workload-socket\") pod \"e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:50.241559 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.241480 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55051609-ecc0-4faf-86ac-d4b04ce9dce5-credential-socket\") pod \"e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:50.241559 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.241517 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55051609-ecc0-4faf-86ac-d4b04ce9dce5-istio-data\") pod \"e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:50.241559 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.241548 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55051609-ecc0-4faf-86ac-d4b04ce9dce5-istio-envoy\") pod \"e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:50.241884 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.241571 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55051609-ecc0-4faf-86ac-d4b04ce9dce5-istiod-ca-cert\") pod \"e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:50.241884 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.241797 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55051609-ecc0-4faf-86ac-d4b04ce9dce5-workload-socket\") pod \"e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:50.241884 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.241824 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55051609-ecc0-4faf-86ac-d4b04ce9dce5-workload-certs\") pod \"e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:50.242061 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.241904 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55051609-ecc0-4faf-86ac-d4b04ce9dce5-istio-data\") pod \"e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:50.242061 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.241950 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55051609-ecc0-4faf-86ac-d4b04ce9dce5-credential-socket\") pod \"e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:50.242365 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.242342 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55051609-ecc0-4faf-86ac-d4b04ce9dce5-istiod-ca-cert\") pod \"e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:50.243913 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.243890 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55051609-ecc0-4faf-86ac-d4b04ce9dce5-istio-envoy\") pod \"e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:50.244044 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.244026 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55051609-ecc0-4faf-86ac-d4b04ce9dce5-istio-podinfo\") pod \"e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:50.256986 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.256920 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55051609-ecc0-4faf-86ac-d4b04ce9dce5-istio-token\") pod \"e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:50.257117 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.257032 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrdrs\" (UniqueName: \"kubernetes.io/projected/55051609-ecc0-4faf-86ac-d4b04ce9dce5-kube-api-access-xrdrs\") pod \"e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:50.318727 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.318663 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:50.479263 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.479228 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc"] Jul 06 12:03:50.480615 ip-10-0-131-176 kubenswrapper[2568]: W0706 12:03:50.480576 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55051609_ecc0_4faf_86ac_d4b04ce9dce5.slice/crio-97d2e30a5b38e338aaa8c5d45a72ec628e911d9212490d3563a70da565c08b62 WatchSource:0}: Error finding container 97d2e30a5b38e338aaa8c5d45a72ec628e911d9212490d3563a70da565c08b62: Status 404 returned error can't find the container with id 97d2e30a5b38e338aaa8c5d45a72ec628e911d9212490d3563a70da565c08b62 Jul 06 12:03:50.482920 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.482885 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 12:03:50.482986 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.482955 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 12:03:50.483025 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.482988 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 12:03:50.928931 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.928896 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" event={"ID":"55051609-ecc0-4faf-86ac-d4b04ce9dce5","Type":"ContainerStarted","Data":"9a115aec4644f9245c9367c6e5450f13e8f6f0da935ce5fd68e392b1117ed971"} Jul 06 12:03:50.929185 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.928963 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" event={"ID":"55051609-ecc0-4faf-86ac-d4b04ce9dce5","Type":"ContainerStarted","Data":"97d2e30a5b38e338aaa8c5d45a72ec628e911d9212490d3563a70da565c08b62"} Jul 06 12:03:50.953639 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:50.953582 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" podStartSLOduration=1.953563113 podStartE2EDuration="1.953563113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:03:50.951707446 +0000 UTC m=+2129.431506463" watchObservedRunningTime="2026-07-06 12:03:50.953563113 +0000 UTC m=+2129.433362496" Jul 06 12:03:51.319820 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:51.319730 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:51.324885 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:51.324858 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:51.935903 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:51.935868 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:51.937192 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:51.937167 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:03:53.014265 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:53.014230 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt"] Jul 06 12:03:53.017964 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:53.017938 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt" Jul 06 12:03:53.021035 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:53.021012 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-8b7fb6-serving-cert\"" Jul 06 12:03:53.027774 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:53.027739 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt"] Jul 06 12:03:53.172474 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:53.172428 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/754dd3ca-8751-4937-9280-dcb7ed56c5f3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt\" (UID: \"754dd3ca-8751-4937-9280-dcb7ed56c5f3\") " pod="opendatahub/maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt" Jul 06 12:03:53.172666 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:53.172505 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gxlf\" (UniqueName: \"kubernetes.io/projected/754dd3ca-8751-4937-9280-dcb7ed56c5f3-kube-api-access-2gxlf\") pod \"maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt\" (UID: \"754dd3ca-8751-4937-9280-dcb7ed56c5f3\") " pod="opendatahub/maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt" Jul 06 12:03:53.172666 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:53.172535 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/754dd3ca-8751-4937-9280-dcb7ed56c5f3-maas-api-tls\") pod \"maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt\" (UID: \"754dd3ca-8751-4937-9280-dcb7ed56c5f3\") " pod="opendatahub/maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt" Jul 06 12:03:53.273431 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:53.273321 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/754dd3ca-8751-4937-9280-dcb7ed56c5f3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt\" (UID: \"754dd3ca-8751-4937-9280-dcb7ed56c5f3\") " pod="opendatahub/maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt" Jul 06 12:03:53.273431 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:53.273409 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gxlf\" (UniqueName: \"kubernetes.io/projected/754dd3ca-8751-4937-9280-dcb7ed56c5f3-kube-api-access-2gxlf\") pod \"maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt\" (UID: \"754dd3ca-8751-4937-9280-dcb7ed56c5f3\") " pod="opendatahub/maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt" Jul 06 12:03:53.273714 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:53.273454 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/754dd3ca-8751-4937-9280-dcb7ed56c5f3-maas-api-tls\") pod \"maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt\" (UID: \"754dd3ca-8751-4937-9280-dcb7ed56c5f3\") " pod="opendatahub/maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt" Jul 06 12:03:53.273714 ip-10-0-131-176 kubenswrapper[2568]: E0706 12:03:53.273576 2568 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-a-8b7fb6-serving-cert: secret "maas-api-e2e-shared-a-8b7fb6-serving-cert" not found Jul 06 12:03:53.273714 ip-10-0-131-176 kubenswrapper[2568]: E0706 12:03:53.273642 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/754dd3ca-8751-4937-9280-dcb7ed56c5f3-maas-api-tls podName:754dd3ca-8751-4937-9280-dcb7ed56c5f3 nodeName:}" failed. No retries permitted until 2026-07-06 12:03:53.773621971 +0000 UTC m=+2132.253420970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/754dd3ca-8751-4937-9280-dcb7ed56c5f3-maas-api-tls") pod "maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt" (UID: "754dd3ca-8751-4937-9280-dcb7ed56c5f3") : secret "maas-api-e2e-shared-a-8b7fb6-serving-cert" not found Jul 06 12:03:53.274035 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:53.274010 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/754dd3ca-8751-4937-9280-dcb7ed56c5f3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt\" (UID: \"754dd3ca-8751-4937-9280-dcb7ed56c5f3\") " pod="opendatahub/maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt" Jul 06 12:03:53.282488 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:53.282462 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gxlf\" (UniqueName: \"kubernetes.io/projected/754dd3ca-8751-4937-9280-dcb7ed56c5f3-kube-api-access-2gxlf\") pod \"maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt\" (UID: \"754dd3ca-8751-4937-9280-dcb7ed56c5f3\") " pod="opendatahub/maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt" Jul 06 12:03:53.779158 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:53.779114 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/754dd3ca-8751-4937-9280-dcb7ed56c5f3-maas-api-tls\") pod \"maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt\" (UID: \"754dd3ca-8751-4937-9280-dcb7ed56c5f3\") " pod="opendatahub/maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt" Jul 06 12:03:53.781715 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:53.781660 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/754dd3ca-8751-4937-9280-dcb7ed56c5f3-maas-api-tls\") pod \"maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt\" (UID: \"754dd3ca-8751-4937-9280-dcb7ed56c5f3\") " pod="opendatahub/maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt" Jul 06 12:03:53.932716 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:53.932664 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt" Jul 06 12:03:54.100254 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:54.100225 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt"] Jul 06 12:03:54.102148 ip-10-0-131-176 kubenswrapper[2568]: W0706 12:03:54.102119 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod754dd3ca_8751_4937_9280_dcb7ed56c5f3.slice/crio-0a43153a67492a4b0f2f5111d379524e727d380ba6720dd599856b8414cc6cff WatchSource:0}: Error finding container 0a43153a67492a4b0f2f5111d379524e727d380ba6720dd599856b8414cc6cff: Status 404 returned error can't find the container with id 0a43153a67492a4b0f2f5111d379524e727d380ba6720dd599856b8414cc6cff Jul 06 12:03:54.955689 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:54.955639 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt" event={"ID":"754dd3ca-8751-4937-9280-dcb7ed56c5f3","Type":"ContainerStarted","Data":"31d39d553f930c2a51b3a789a4ff825d8cb6eccae8f6975adc7f32d3e4979fd1"} Jul 06 12:03:54.955900 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:54.955710 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt" event={"ID":"754dd3ca-8751-4937-9280-dcb7ed56c5f3","Type":"ContainerStarted","Data":"0a43153a67492a4b0f2f5111d379524e727d380ba6720dd599856b8414cc6cff"} Jul 06 12:03:54.955900 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:03:54.955736 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt" Jul 06 12:04:00.965954 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:00.965922 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt" Jul 06 12:04:00.988502 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:00.988433 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt" podStartSLOduration=8.685599998 podStartE2EDuration="8.988414853s" totalImagesPullingTime="302.814855ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:03:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:03:54.972823548 +0000 UTC m=+2133.452622565" watchObservedRunningTime="2026-07-06 12:04:00.988414853 +0000 UTC m=+2139.468213901" Jul 06 12:04:01.239893 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:01.239784 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8"] Jul 06 12:04:01.247101 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:01.245619 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8" Jul 06 12:04:01.249132 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:01.248903 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-cf8c6a-serving-cert\"" Jul 06 12:04:01.259703 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:01.259653 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8"] Jul 06 12:04:01.351319 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:01.351275 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8\" (UID: \"7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7\") " pod="opendatahub/maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8" Jul 06 12:04:01.351659 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:01.351631 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7-maas-api-tls\") pod \"maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8\" (UID: \"7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7\") " pod="opendatahub/maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8" Jul 06 12:04:01.351862 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:01.351839 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhf5k\" (UniqueName: \"kubernetes.io/projected/7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7-kube-api-access-bhf5k\") pod \"maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8\" (UID: \"7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7\") " pod="opendatahub/maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8" Jul 06 12:04:01.453472 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:01.453432 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7-maas-api-tls\") pod \"maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8\" (UID: \"7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7\") " pod="opendatahub/maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8" Jul 06 12:04:01.453773 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:01.453751 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhf5k\" (UniqueName: \"kubernetes.io/projected/7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7-kube-api-access-bhf5k\") pod \"maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8\" (UID: \"7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7\") " pod="opendatahub/maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8" Jul 06 12:04:01.453989 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:01.453972 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8\" (UID: \"7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7\") " pod="opendatahub/maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8" Jul 06 12:04:01.454964 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:01.454938 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8\" (UID: \"7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7\") " pod="opendatahub/maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8" Jul 06 12:04:01.458284 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:01.458259 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7-maas-api-tls\") pod \"maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8\" (UID: \"7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7\") " pod="opendatahub/maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8" Jul 06 12:04:01.463696 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:01.463655 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhf5k\" (UniqueName: \"kubernetes.io/projected/7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7-kube-api-access-bhf5k\") pod \"maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8\" (UID: \"7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7\") " pod="opendatahub/maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8" Jul 06 12:04:01.568630 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:01.568534 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8" Jul 06 12:04:01.760280 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:01.760236 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8"] Jul 06 12:04:01.766825 ip-10-0-131-176 kubenswrapper[2568]: W0706 12:04:01.766795 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca2cd4f_6ecc_4e7a_804b_4c5401dbd8a7.slice/crio-1a2cf6e682f6e4a14f89580d05fa640d96d7fc5d63f44128b6f18687924d0df6 WatchSource:0}: Error finding container 1a2cf6e682f6e4a14f89580d05fa640d96d7fc5d63f44128b6f18687924d0df6: Status 404 returned error can't find the container with id 1a2cf6e682f6e4a14f89580d05fa640d96d7fc5d63f44128b6f18687924d0df6 Jul 06 12:04:01.995235 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:01.995196 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8" event={"ID":"7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7","Type":"ContainerStarted","Data":"1a2cf6e682f6e4a14f89580d05fa640d96d7fc5d63f44128b6f18687924d0df6"} Jul 06 12:04:03.001759 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:03.001718 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8" event={"ID":"7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7","Type":"ContainerStarted","Data":"a2654483012c833f176b47c6b1f5ad84f1542ecd3ca3f9b9419dff5d5edca4fc"} Jul 06 12:04:03.002167 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:03.001812 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8" Jul 06 12:04:03.021997 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:03.021932 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8" podStartSLOduration=1.59992986 podStartE2EDuration="2.021914528s" totalImagesPullingTime="421.984668ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 12:04:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:04:03.019099244 +0000 UTC m=+2141.498898260" watchObservedRunningTime="2026-07-06 12:04:03.021914528 +0000 UTC m=+2141.501713543" Jul 06 12:04:07.977880 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:07.977834 2568 patch_prober.go:28] interesting pod/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 06 12:04:07.978297 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:07.977915 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" podUID="55051609-ecc0-4faf-86ac-d4b04ce9dce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 12:04:09.012707 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:09.012660 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8" Jul 06 12:04:09.041666 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:09.041634 2568 generic.go:356] "Generic (PLEG): container finished" podID="55051609-ecc0-4faf-86ac-d4b04ce9dce5" containerID="9a115aec4644f9245c9367c6e5450f13e8f6f0da935ce5fd68e392b1117ed971" exitCode=0 Jul 06 12:04:09.041874 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:09.041713 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" event={"ID":"55051609-ecc0-4faf-86ac-d4b04ce9dce5","Type":"ContainerDied","Data":"9a115aec4644f9245c9367c6e5450f13e8f6f0da935ce5fd68e392b1117ed971"} Jul 06 12:04:09.042255 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:09.042238 2568 scope.go:117] "RemoveContainer" containerID="9a115aec4644f9245c9367c6e5450f13e8f6f0da935ce5fd68e392b1117ed971" Jul 06 12:04:09.043552 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:09.043519 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 12:04:09.043652 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:09.043605 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 12:04:09.043730 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:09.043653 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 12:04:10.068954 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:10.068915 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" event={"ID":"55051609-ecc0-4faf-86ac-d4b04ce9dce5","Type":"ContainerStarted","Data":"262e0da509fb65326290411b485fc2729552c55a01f96f8f083ebd14b301ec05"} Jul 06 12:04:10.319026 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:10.318937 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:04:10.319186 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:10.319156 2568 patch_prober.go:28] interesting pod/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 06 12:04:10.319229 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:10.319203 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" podUID="55051609-ecc0-4faf-86ac-d4b04ce9dce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 06 12:04:11.319780 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:11.319750 2568 patch_prober.go:28] interesting pod/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 06 12:04:11.320230 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:11.319892 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" podUID="55051609-ecc0-4faf-86ac-d4b04ce9dce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 06 12:04:12.319773 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:12.319740 2568 patch_prober.go:28] interesting pod/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 06 12:04:12.319957 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:12.319797 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" podUID="55051609-ecc0-4faf-86ac-d4b04ce9dce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 06 12:04:13.319976 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:13.319941 2568 patch_prober.go:28] interesting pod/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 06 12:04:13.320439 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:13.320005 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" podUID="55051609-ecc0-4faf-86ac-d4b04ce9dce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 06 12:04:14.323093 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:14.323007 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:04:14.323491 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:14.323296 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:04:14.324209 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:04:14.324186 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:10:17.753505 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:10:17.753412 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt"] Jul 06 12:10:17.754037 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:10:17.753656 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt" podUID="754dd3ca-8751-4937-9280-dcb7ed56c5f3" containerName="maas-api" containerID="cri-o://31d39d553f930c2a51b3a789a4ff825d8cb6eccae8f6975adc7f32d3e4979fd1" gracePeriod=30 Jul 06 12:10:18.001092 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:10:18.001065 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt" Jul 06 12:10:18.042456 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:10:18.042372 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gxlf\" (UniqueName: \"kubernetes.io/projected/754dd3ca-8751-4937-9280-dcb7ed56c5f3-kube-api-access-2gxlf\") pod \"754dd3ca-8751-4937-9280-dcb7ed56c5f3\" (UID: \"754dd3ca-8751-4937-9280-dcb7ed56c5f3\") " Jul 06 12:10:18.042456 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:10:18.042404 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/754dd3ca-8751-4937-9280-dcb7ed56c5f3-maas-api-tls\") pod \"754dd3ca-8751-4937-9280-dcb7ed56c5f3\" (UID: \"754dd3ca-8751-4937-9280-dcb7ed56c5f3\") " Jul 06 12:10:18.042696 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:10:18.042492 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/754dd3ca-8751-4937-9280-dcb7ed56c5f3-openshift-service-ca\") pod \"754dd3ca-8751-4937-9280-dcb7ed56c5f3\" (UID: \"754dd3ca-8751-4937-9280-dcb7ed56c5f3\") " Jul 06 12:10:18.042899 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:10:18.042874 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/754dd3ca-8751-4937-9280-dcb7ed56c5f3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "754dd3ca-8751-4937-9280-dcb7ed56c5f3" (UID: "754dd3ca-8751-4937-9280-dcb7ed56c5f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 12:10:18.044554 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:10:18.044521 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/754dd3ca-8751-4937-9280-dcb7ed56c5f3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "754dd3ca-8751-4937-9280-dcb7ed56c5f3" (UID: "754dd3ca-8751-4937-9280-dcb7ed56c5f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:10:18.044662 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:10:18.044552 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/754dd3ca-8751-4937-9280-dcb7ed56c5f3-kube-api-access-2gxlf" (OuterVolumeSpecName: "kube-api-access-2gxlf") pod "754dd3ca-8751-4937-9280-dcb7ed56c5f3" (UID: "754dd3ca-8751-4937-9280-dcb7ed56c5f3"). InnerVolumeSpecName "kube-api-access-2gxlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:10:18.080739 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:10:18.080708 2568 generic.go:356] "Generic (PLEG): container finished" podID="754dd3ca-8751-4937-9280-dcb7ed56c5f3" containerID="31d39d553f930c2a51b3a789a4ff825d8cb6eccae8f6975adc7f32d3e4979fd1" exitCode=0 Jul 06 12:10:18.080835 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:10:18.080797 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt" Jul 06 12:10:18.080886 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:10:18.080840 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt" event={"ID":"754dd3ca-8751-4937-9280-dcb7ed56c5f3","Type":"ContainerDied","Data":"31d39d553f930c2a51b3a789a4ff825d8cb6eccae8f6975adc7f32d3e4979fd1"} Jul 06 12:10:18.080922 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:10:18.080890 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt" event={"ID":"754dd3ca-8751-4937-9280-dcb7ed56c5f3","Type":"ContainerDied","Data":"0a43153a67492a4b0f2f5111d379524e727d380ba6720dd599856b8414cc6cff"} Jul 06 12:10:18.080922 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:10:18.080917 2568 scope.go:117] "RemoveContainer" containerID="31d39d553f930c2a51b3a789a4ff825d8cb6eccae8f6975adc7f32d3e4979fd1" Jul 06 12:10:18.089641 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:10:18.089626 2568 scope.go:117] "RemoveContainer" containerID="31d39d553f930c2a51b3a789a4ff825d8cb6eccae8f6975adc7f32d3e4979fd1" Jul 06 12:10:18.089924 ip-10-0-131-176 kubenswrapper[2568]: E0706 12:10:18.089906 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31d39d553f930c2a51b3a789a4ff825d8cb6eccae8f6975adc7f32d3e4979fd1\": container with ID starting with 31d39d553f930c2a51b3a789a4ff825d8cb6eccae8f6975adc7f32d3e4979fd1 not found: ID does not exist" containerID="31d39d553f930c2a51b3a789a4ff825d8cb6eccae8f6975adc7f32d3e4979fd1" Jul 06 12:10:18.089976 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:10:18.089933 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31d39d553f930c2a51b3a789a4ff825d8cb6eccae8f6975adc7f32d3e4979fd1"} err="failed to get container status \"31d39d553f930c2a51b3a789a4ff825d8cb6eccae8f6975adc7f32d3e4979fd1\": rpc error: code = NotFound desc = could not find container \"31d39d553f930c2a51b3a789a4ff825d8cb6eccae8f6975adc7f32d3e4979fd1\": container with ID starting with 31d39d553f930c2a51b3a789a4ff825d8cb6eccae8f6975adc7f32d3e4979fd1 not found: ID does not exist" Jul 06 12:10:18.099392 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:10:18.099372 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt"] Jul 06 12:10:18.104304 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:10:18.104284 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-8b7fb6-7f5fbf5d7f-cbqgt"] Jul 06 12:10:18.143974 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:10:18.143937 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/754dd3ca-8751-4937-9280-dcb7ed56c5f3-openshift-service-ca\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:10:18.143974 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:10:18.143972 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gxlf\" (UniqueName: \"kubernetes.io/projected/754dd3ca-8751-4937-9280-dcb7ed56c5f3-kube-api-access-2gxlf\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:10:18.143974 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:10:18.143984 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/754dd3ca-8751-4937-9280-dcb7ed56c5f3-maas-api-tls\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:10:20.053082 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:10:20.053045 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="754dd3ca-8751-4937-9280-dcb7ed56c5f3" path="/var/lib/kubelet/pods/754dd3ca-8751-4937-9280-dcb7ed56c5f3/volumes" Jul 06 12:10:59.662878 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:10:59.662840 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc"] Jul 06 12:10:59.663479 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:10:59.663047 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" podUID="55051609-ecc0-4faf-86ac-d4b04ce9dce5" containerName="istio-proxy" containerID="cri-o://262e0da509fb65326290411b485fc2729552c55a01f96f8f083ebd14b301ec05" gracePeriod=30 Jul 06 12:11:00.708661 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:00.708629 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8"] Jul 06 12:11:00.709182 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:00.708873 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8" podUID="7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7" containerName="maas-api" containerID="cri-o://a2654483012c833f176b47c6b1f5ad84f1542ecd3ca3f9b9419dff5d5edca4fc" gracePeriod=30 Jul 06 12:11:00.954559 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:00.954536 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8" Jul 06 12:11:01.010213 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:01.010113 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7-maas-api-tls\") pod \"7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7\" (UID: \"7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7\") " Jul 06 12:11:01.010213 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:01.010176 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhf5k\" (UniqueName: \"kubernetes.io/projected/7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7-kube-api-access-bhf5k\") pod \"7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7\" (UID: \"7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7\") " Jul 06 12:11:01.010501 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:01.010227 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7-openshift-service-ca\") pod \"7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7\" (UID: \"7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7\") " Jul 06 12:11:01.010599 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:01.010577 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7" (UID: "7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 12:11:01.012193 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:01.012172 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7" (UID: "7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 12:11:01.012316 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:01.012296 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7-kube-api-access-bhf5k" (OuterVolumeSpecName: "kube-api-access-bhf5k") pod "7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7" (UID: "7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7"). InnerVolumeSpecName "kube-api-access-bhf5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:11:01.110940 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:01.110907 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7-maas-api-tls\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:11:01.110940 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:01.110934 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhf5k\" (UniqueName: \"kubernetes.io/projected/7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7-kube-api-access-bhf5k\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:11:01.110940 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:01.110946 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7-openshift-service-ca\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:11:01.307966 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:01.307871 2568 generic.go:356] "Generic (PLEG): container finished" podID="7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7" containerID="a2654483012c833f176b47c6b1f5ad84f1542ecd3ca3f9b9419dff5d5edca4fc" exitCode=0 Jul 06 12:11:01.307966 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:01.307919 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8" Jul 06 12:11:01.308233 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:01.307964 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8" event={"ID":"7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7","Type":"ContainerDied","Data":"a2654483012c833f176b47c6b1f5ad84f1542ecd3ca3f9b9419dff5d5edca4fc"} Jul 06 12:11:01.308233 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:01.308029 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8" event={"ID":"7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7","Type":"ContainerDied","Data":"1a2cf6e682f6e4a14f89580d05fa640d96d7fc5d63f44128b6f18687924d0df6"} Jul 06 12:11:01.308233 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:01.308060 2568 scope.go:117] "RemoveContainer" containerID="a2654483012c833f176b47c6b1f5ad84f1542ecd3ca3f9b9419dff5d5edca4fc" Jul 06 12:11:01.317564 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:01.317547 2568 scope.go:117] "RemoveContainer" containerID="a2654483012c833f176b47c6b1f5ad84f1542ecd3ca3f9b9419dff5d5edca4fc" Jul 06 12:11:01.317898 ip-10-0-131-176 kubenswrapper[2568]: E0706 12:11:01.317878 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2654483012c833f176b47c6b1f5ad84f1542ecd3ca3f9b9419dff5d5edca4fc\": container with ID starting with a2654483012c833f176b47c6b1f5ad84f1542ecd3ca3f9b9419dff5d5edca4fc not found: ID does not exist" containerID="a2654483012c833f176b47c6b1f5ad84f1542ecd3ca3f9b9419dff5d5edca4fc" Jul 06 12:11:01.317962 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:01.317907 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2654483012c833f176b47c6b1f5ad84f1542ecd3ca3f9b9419dff5d5edca4fc"} err="failed to get container status \"a2654483012c833f176b47c6b1f5ad84f1542ecd3ca3f9b9419dff5d5edca4fc\": rpc error: code = NotFound desc = could not find container \"a2654483012c833f176b47c6b1f5ad84f1542ecd3ca3f9b9419dff5d5edca4fc\": container with ID starting with a2654483012c833f176b47c6b1f5ad84f1542ecd3ca3f9b9419dff5d5edca4fc not found: ID does not exist" Jul 06 12:11:01.331396 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:01.331369 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8"] Jul 06 12:11:01.335359 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:01.335336 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-cf8c6a-99d6f9684-w8vc8"] Jul 06 12:11:02.051809 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:02.051777 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7" path="/var/lib/kubelet/pods/7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7/volumes" Jul 06 12:11:04.924522 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:04.924462 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:11:04.947374 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:04.947349 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55051609-ecc0-4faf-86ac-d4b04ce9dce5-istio-podinfo\") pod \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " Jul 06 12:11:04.947524 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:04.947404 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55051609-ecc0-4faf-86ac-d4b04ce9dce5-istio-token\") pod \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " Jul 06 12:11:04.947590 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:04.947523 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55051609-ecc0-4faf-86ac-d4b04ce9dce5-workload-socket\") pod \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " Jul 06 12:11:04.947590 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:04.947568 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55051609-ecc0-4faf-86ac-d4b04ce9dce5-workload-certs\") pod \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " Jul 06 12:11:04.947714 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:04.947596 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55051609-ecc0-4faf-86ac-d4b04ce9dce5-istio-data\") pod \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " Jul 06 12:11:04.947714 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:04.947628 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55051609-ecc0-4faf-86ac-d4b04ce9dce5-istiod-ca-cert\") pod \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " Jul 06 12:11:04.947714 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:04.947689 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55051609-ecc0-4faf-86ac-d4b04ce9dce5-credential-socket\") pod \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " Jul 06 12:11:04.947876 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:04.947738 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrdrs\" (UniqueName: \"kubernetes.io/projected/55051609-ecc0-4faf-86ac-d4b04ce9dce5-kube-api-access-xrdrs\") pod \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " Jul 06 12:11:04.947876 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:04.947767 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55051609-ecc0-4faf-86ac-d4b04ce9dce5-istio-envoy\") pod \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\" (UID: \"55051609-ecc0-4faf-86ac-d4b04ce9dce5\") " Jul 06 12:11:04.947876 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:04.947825 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55051609-ecc0-4faf-86ac-d4b04ce9dce5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "55051609-ecc0-4faf-86ac-d4b04ce9dce5" (UID: "55051609-ecc0-4faf-86ac-d4b04ce9dce5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:11:04.947876 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:04.947834 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55051609-ecc0-4faf-86ac-d4b04ce9dce5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "55051609-ecc0-4faf-86ac-d4b04ce9dce5" (UID: "55051609-ecc0-4faf-86ac-d4b04ce9dce5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:11:04.948108 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:04.948076 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55051609-ecc0-4faf-86ac-d4b04ce9dce5-workload-socket\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:11:04.948169 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:04.948110 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55051609-ecc0-4faf-86ac-d4b04ce9dce5-workload-certs\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:11:04.948231 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:04.948177 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55051609-ecc0-4faf-86ac-d4b04ce9dce5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "55051609-ecc0-4faf-86ac-d4b04ce9dce5" (UID: "55051609-ecc0-4faf-86ac-d4b04ce9dce5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 12:11:04.948292 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:04.948231 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55051609-ecc0-4faf-86ac-d4b04ce9dce5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "55051609-ecc0-4faf-86ac-d4b04ce9dce5" (UID: "55051609-ecc0-4faf-86ac-d4b04ce9dce5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:11:04.949064 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:04.949029 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55051609-ecc0-4faf-86ac-d4b04ce9dce5-istio-data" (OuterVolumeSpecName: "istio-data") pod "55051609-ecc0-4faf-86ac-d4b04ce9dce5" (UID: "55051609-ecc0-4faf-86ac-d4b04ce9dce5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:11:04.949979 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:04.949950 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55051609-ecc0-4faf-86ac-d4b04ce9dce5-istio-token" (OuterVolumeSpecName: "istio-token") pod "55051609-ecc0-4faf-86ac-d4b04ce9dce5" (UID: "55051609-ecc0-4faf-86ac-d4b04ce9dce5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:11:04.950090 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:04.949959 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55051609-ecc0-4faf-86ac-d4b04ce9dce5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "55051609-ecc0-4faf-86ac-d4b04ce9dce5" (UID: "55051609-ecc0-4faf-86ac-d4b04ce9dce5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 12:11:04.950184 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:04.950157 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55051609-ecc0-4faf-86ac-d4b04ce9dce5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "55051609-ecc0-4faf-86ac-d4b04ce9dce5" (UID: "55051609-ecc0-4faf-86ac-d4b04ce9dce5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 12:11:04.950284 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:04.950197 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55051609-ecc0-4faf-86ac-d4b04ce9dce5-kube-api-access-xrdrs" (OuterVolumeSpecName: "kube-api-access-xrdrs") pod "55051609-ecc0-4faf-86ac-d4b04ce9dce5" (UID: "55051609-ecc0-4faf-86ac-d4b04ce9dce5"). InnerVolumeSpecName "kube-api-access-xrdrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 12:11:05.049492 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:05.049464 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55051609-ecc0-4faf-86ac-d4b04ce9dce5-istio-podinfo\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:11:05.049492 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:05.049490 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55051609-ecc0-4faf-86ac-d4b04ce9dce5-istio-token\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:11:05.049492 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:05.049500 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55051609-ecc0-4faf-86ac-d4b04ce9dce5-istio-data\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:11:05.049726 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:05.049507 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55051609-ecc0-4faf-86ac-d4b04ce9dce5-istiod-ca-cert\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:11:05.049726 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:05.049516 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55051609-ecc0-4faf-86ac-d4b04ce9dce5-credential-socket\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:11:05.049726 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:05.049525 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrdrs\" (UniqueName: \"kubernetes.io/projected/55051609-ecc0-4faf-86ac-d4b04ce9dce5-kube-api-access-xrdrs\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:11:05.049726 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:05.049534 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55051609-ecc0-4faf-86ac-d4b04ce9dce5-istio-envoy\") on node \"ip-10-0-131-176.ec2.internal\" DevicePath \"\"" Jul 06 12:11:05.334333 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:05.334243 2568 generic.go:356] "Generic (PLEG): container finished" podID="55051609-ecc0-4faf-86ac-d4b04ce9dce5" containerID="262e0da509fb65326290411b485fc2729552c55a01f96f8f083ebd14b301ec05" exitCode=0 Jul 06 12:11:05.334333 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:05.334310 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" Jul 06 12:11:05.334514 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:05.334334 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" event={"ID":"55051609-ecc0-4faf-86ac-d4b04ce9dce5","Type":"ContainerDied","Data":"262e0da509fb65326290411b485fc2729552c55a01f96f8f083ebd14b301ec05"} Jul 06 12:11:05.334514 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:05.334402 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc" event={"ID":"55051609-ecc0-4faf-86ac-d4b04ce9dce5","Type":"ContainerDied","Data":"97d2e30a5b38e338aaa8c5d45a72ec628e911d9212490d3563a70da565c08b62"} Jul 06 12:11:05.334514 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:05.334431 2568 scope.go:117] "RemoveContainer" containerID="262e0da509fb65326290411b485fc2729552c55a01f96f8f083ebd14b301ec05" Jul 06 12:11:05.344065 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:05.344043 2568 scope.go:117] "RemoveContainer" containerID="9a115aec4644f9245c9367c6e5450f13e8f6f0da935ce5fd68e392b1117ed971" Jul 06 12:11:05.352344 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:05.352322 2568 scope.go:117] "RemoveContainer" containerID="262e0da509fb65326290411b485fc2729552c55a01f96f8f083ebd14b301ec05" Jul 06 12:11:05.352611 ip-10-0-131-176 kubenswrapper[2568]: E0706 12:11:05.352592 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"262e0da509fb65326290411b485fc2729552c55a01f96f8f083ebd14b301ec05\": container with ID starting with 262e0da509fb65326290411b485fc2729552c55a01f96f8f083ebd14b301ec05 not found: ID does not exist" containerID="262e0da509fb65326290411b485fc2729552c55a01f96f8f083ebd14b301ec05" Jul 06 12:11:05.352665 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:05.352619 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"262e0da509fb65326290411b485fc2729552c55a01f96f8f083ebd14b301ec05"} err="failed to get container status \"262e0da509fb65326290411b485fc2729552c55a01f96f8f083ebd14b301ec05\": rpc error: code = NotFound desc = could not find container \"262e0da509fb65326290411b485fc2729552c55a01f96f8f083ebd14b301ec05\": container with ID starting with 262e0da509fb65326290411b485fc2729552c55a01f96f8f083ebd14b301ec05 not found: ID does not exist" Jul 06 12:11:05.352665 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:05.352635 2568 scope.go:117] "RemoveContainer" containerID="9a115aec4644f9245c9367c6e5450f13e8f6f0da935ce5fd68e392b1117ed971" Jul 06 12:11:05.352858 ip-10-0-131-176 kubenswrapper[2568]: E0706 12:11:05.352843 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a115aec4644f9245c9367c6e5450f13e8f6f0da935ce5fd68e392b1117ed971\": container with ID starting with 9a115aec4644f9245c9367c6e5450f13e8f6f0da935ce5fd68e392b1117ed971 not found: ID does not exist" containerID="9a115aec4644f9245c9367c6e5450f13e8f6f0da935ce5fd68e392b1117ed971" Jul 06 12:11:05.352905 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:05.352862 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a115aec4644f9245c9367c6e5450f13e8f6f0da935ce5fd68e392b1117ed971"} err="failed to get container status \"9a115aec4644f9245c9367c6e5450f13e8f6f0da935ce5fd68e392b1117ed971\": rpc error: code = NotFound desc = could not find container \"9a115aec4644f9245c9367c6e5450f13e8f6f0da935ce5fd68e392b1117ed971\": container with ID starting with 9a115aec4644f9245c9367c6e5450f13e8f6f0da935ce5fd68e392b1117ed971 not found: ID does not exist" Jul 06 12:11:05.359445 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:05.359412 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc"] Jul 06 12:11:05.363257 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:05.363235 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-8b7fb6-openshift-default-7666795d8c-jszqc"] Jul 06 12:11:06.051715 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:06.051642 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55051609-ecc0-4faf-86ac-d4b04ce9dce5" path="/var/lib/kubelet/pods/55051609-ecc0-4faf-86ac-d4b04ce9dce5/volumes" Jul 06 12:11:11.890738 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:11.890701 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-gh8vj_609ac905-0773-4403-8e9b-8fe8a4dcf908/manager/0.log" Jul 06 12:11:12.014126 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:12.014100 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-6zprf_4db87dba-c23e-4a77-b58b-b9de0644d486/manager/0.log" Jul 06 12:11:12.150940 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:12.150860 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-ccdfff8b-g5tjf_16d80f49-fc68-452e-ab42-eb4dd72a1dfe/maas-api/0.log" Jul 06 12:11:12.268794 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:12.268761 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722305-dwsj4_a5604901-4af8-41b7-be0e-92cc18b812b1/cleanup/0.log" Jul 06 12:11:12.388576 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:12.388544 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722320-rlln9_bf2357b0-6395-4bb3-8bfd-6099d105a418/cleanup/0.log" Jul 06 12:11:12.512699 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:12.512605 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5fc875b447-dlqtv_27876b93-9b29-42c7-b21e-902222ff3daa/manager/0.log" Jul 06 12:11:12.625183 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:12.625155 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-95wlx_dfcbe27d-5c42-45f9-a719-5b43b710c11d/server/0.log" Jul 06 12:11:12.747921 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:12.747886 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-kdsrp_6bfe9a65-bdfc-431b-b376-40f507daf37a/manager/0.log" Jul 06 12:11:13.002002 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:13.001973 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cb7946974-kcmtw_ce314f78-7470-45c3-9a73-cede06d44b62/copy-manifests/0.log" Jul 06 12:11:13.022472 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:13.022440 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cb7946974-kcmtw_ce314f78-7470-45c3-9a73-cede06d44b62/manager/0.log" Jul 06 12:11:14.035729 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:14.035695 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx_969a11c9-a0fa-45d1-b093-4bb6c5fc84b4/pull/0.log" Jul 06 12:11:14.049614 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:14.049589 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx_969a11c9-a0fa-45d1-b093-4bb6c5fc84b4/extract/0.log" Jul 06 12:11:14.061471 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:14.061442 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx_969a11c9-a0fa-45d1-b093-4bb6c5fc84b4/util/0.log" Jul 06 12:11:14.175561 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:14.175534 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql_eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540/util/0.log" Jul 06 12:11:14.184919 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:14.184888 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql_eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540/pull/0.log" Jul 06 12:11:14.194317 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:14.194300 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql_eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540/extract/0.log" Jul 06 12:11:14.306645 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:14.306558 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf_6f247d94-5720-4a5e-af09-61639affce37/util/0.log" Jul 06 12:11:14.318836 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:14.318811 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf_6f247d94-5720-4a5e-af09-61639affce37/pull/0.log" Jul 06 12:11:14.330722 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:14.330702 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf_6f247d94-5720-4a5e-af09-61639affce37/extract/0.log" Jul 06 12:11:14.441695 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:14.441651 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb_a6085760-c8a0-4f8c-9494-e86fd58675ac/util/0.log" Jul 06 12:11:14.448878 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:14.448856 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb_a6085760-c8a0-4f8c-9494-e86fd58675ac/pull/0.log" Jul 06 12:11:14.461523 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:14.461498 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb_a6085760-c8a0-4f8c-9494-e86fd58675ac/extract/0.log" Jul 06 12:11:14.826629 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:14.826592 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-ptbjr_8565fb29-137e-4199-b150-dc34077ab6b8/manager/0.log" Jul 06 12:11:15.206841 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:15.206810 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-vtfn2_63b1cd37-c490-410b-92bd-be6d823a4efe/manager/0.log" Jul 06 12:11:15.790154 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:15.790126 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678n6qkn_b9859ff5-1763-4c0c-a074-3e9f86a65f68/istio-proxy/0.log" Jul 06 12:11:15.910877 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:15.910847 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-56gcm_e2c53d85-1c9b-4fba-a7b1-05cff745478b/discovery/0.log" Jul 06 12:11:16.027902 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:16.027870 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64c57c544d-4nq7x_a7dbffb2-b3ec-4aea-a451-ef4ad10ab5f4/kube-auth-proxy/0.log" Jul 06 12:11:16.971539 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:16.971513 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-556f6_746dfe16-0774-46e0-848d-3652274bba02/main/0.log" Jul 06 12:11:17.319349 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:17.319272 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgv4z8_3e3c1fdd-0b43-4bd8-b498-8a3f0df14bef/main/0.log" Jul 06 12:11:24.888748 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:24.888714 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9mqf5_9c4fa664-243d-4a56-91ce-a58f01d77768/global-pull-secret-syncer/0.log" Jul 06 12:11:25.113018 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:25.112986 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-l89bb_218f64a8-8fd8-462b-972d-ff9ecedb6a74/konnectivity-agent/0.log" Jul 06 12:11:25.136205 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:25.136178 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-176.ec2.internal_cf4e10fd22892d39b2d489b025b34f3d/haproxy/0.log" Jul 06 12:11:29.560331 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:29.560300 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx_969a11c9-a0fa-45d1-b093-4bb6c5fc84b4/extract/0.log" Jul 06 12:11:29.581862 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:29.581838 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx_969a11c9-a0fa-45d1-b093-4bb6c5fc84b4/util/0.log" Jul 06 12:11:29.606250 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:29.606225 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d7rsx_969a11c9-a0fa-45d1-b093-4bb6c5fc84b4/pull/0.log" Jul 06 12:11:29.632956 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:29.632933 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql_eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540/extract/0.log" Jul 06 12:11:29.656304 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:29.656281 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql_eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540/util/0.log" Jul 06 12:11:29.681692 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:29.681648 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jrtql_eda6f93f-8f1d-49ba-a3dd-7e1cc64f1540/pull/0.log" Jul 06 12:11:29.707658 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:29.707627 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf_6f247d94-5720-4a5e-af09-61639affce37/extract/0.log" Jul 06 12:11:29.742648 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:29.742626 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf_6f247d94-5720-4a5e-af09-61639affce37/util/0.log" Jul 06 12:11:29.759055 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:29.759023 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l7vdf_6f247d94-5720-4a5e-af09-61639affce37/pull/0.log" Jul 06 12:11:29.786081 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:29.786062 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb_a6085760-c8a0-4f8c-9494-e86fd58675ac/extract/0.log" Jul 06 12:11:29.807612 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:29.807589 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb_a6085760-c8a0-4f8c-9494-e86fd58675ac/util/0.log" Jul 06 12:11:29.830649 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:29.830591 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1d5srb_a6085760-c8a0-4f8c-9494-e86fd58675ac/pull/0.log" Jul 06 12:11:30.310100 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:30.310070 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-ptbjr_8565fb29-137e-4199-b150-dc34077ab6b8/manager/0.log" Jul 06 12:11:30.586961 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:30.586855 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-vtfn2_63b1cd37-c490-410b-92bd-be6d823a4efe/manager/0.log" Jul 06 12:11:32.550339 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:32.550295 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-jjrll_6e16731d-a528-49c9-8998-dcc1d0b44957/cluster-monitoring-operator/0.log" Jul 06 12:11:32.871384 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:32.871298 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n4nfk_3bcac4b6-e83b-48d0-aacd-76e54565ac0d/node-exporter/0.log" Jul 06 12:11:32.891733 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:32.891711 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n4nfk_3bcac4b6-e83b-48d0-aacd-76e54565ac0d/kube-rbac-proxy/0.log" Jul 06 12:11:32.918394 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:32.918373 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n4nfk_3bcac4b6-e83b-48d0-aacd-76e54565ac0d/init-textfile/0.log" Jul 06 12:11:34.410025 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.409985 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qzjtz/perf-node-gather-daemonset-r5tst"] Jul 06 12:11:34.410425 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.410374 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55051609-ecc0-4faf-86ac-d4b04ce9dce5" containerName="istio-proxy" Jul 06 12:11:34.410425 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.410388 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="55051609-ecc0-4faf-86ac-d4b04ce9dce5" containerName="istio-proxy" Jul 06 12:11:34.410425 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.410404 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7" containerName="maas-api" Jul 06 12:11:34.410425 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.410410 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7" containerName="maas-api" Jul 06 12:11:34.410425 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.410423 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55051609-ecc0-4faf-86ac-d4b04ce9dce5" containerName="istio-proxy" Jul 06 12:11:34.410425 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.410429 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="55051609-ecc0-4faf-86ac-d4b04ce9dce5" containerName="istio-proxy" Jul 06 12:11:34.410702 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.410439 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="754dd3ca-8751-4937-9280-dcb7ed56c5f3" containerName="maas-api" Jul 06 12:11:34.410702 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.410445 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="754dd3ca-8751-4937-9280-dcb7ed56c5f3" containerName="maas-api" Jul 06 12:11:34.410702 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.410503 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ca2cd4f-6ecc-4e7a-804b-4c5401dbd8a7" containerName="maas-api" Jul 06 12:11:34.410702 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.410512 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="55051609-ecc0-4faf-86ac-d4b04ce9dce5" containerName="istio-proxy" Jul 06 12:11:34.410702 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.410519 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="754dd3ca-8751-4937-9280-dcb7ed56c5f3" containerName="maas-api" Jul 06 12:11:34.413759 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.413739 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qzjtz/perf-node-gather-daemonset-r5tst" Jul 06 12:11:34.416460 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.416436 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qzjtz\"/\"default-dockercfg-jkctc\"" Jul 06 12:11:34.416460 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.416457 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qzjtz\"/\"openshift-service-ca.crt\"" Jul 06 12:11:34.416615 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.416477 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qzjtz\"/\"kube-root-ca.crt\"" Jul 06 12:11:34.424331 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.424306 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qzjtz/perf-node-gather-daemonset-r5tst"] Jul 06 12:11:34.613218 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.613185 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkhtn\" (UniqueName: \"kubernetes.io/projected/8a7203a7-e346-47b2-8df1-54f9d2e64d85-kube-api-access-gkhtn\") pod \"perf-node-gather-daemonset-r5tst\" (UID: \"8a7203a7-e346-47b2-8df1-54f9d2e64d85\") " pod="openshift-must-gather-qzjtz/perf-node-gather-daemonset-r5tst" Jul 06 12:11:34.613218 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.613220 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8a7203a7-e346-47b2-8df1-54f9d2e64d85-podres\") pod \"perf-node-gather-daemonset-r5tst\" (UID: \"8a7203a7-e346-47b2-8df1-54f9d2e64d85\") " pod="openshift-must-gather-qzjtz/perf-node-gather-daemonset-r5tst" Jul 06 12:11:34.613467 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.613252 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a7203a7-e346-47b2-8df1-54f9d2e64d85-lib-modules\") pod \"perf-node-gather-daemonset-r5tst\" (UID: \"8a7203a7-e346-47b2-8df1-54f9d2e64d85\") " pod="openshift-must-gather-qzjtz/perf-node-gather-daemonset-r5tst" Jul 06 12:11:34.613467 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.613357 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a7203a7-e346-47b2-8df1-54f9d2e64d85-sys\") pod \"perf-node-gather-daemonset-r5tst\" (UID: \"8a7203a7-e346-47b2-8df1-54f9d2e64d85\") " pod="openshift-must-gather-qzjtz/perf-node-gather-daemonset-r5tst" Jul 06 12:11:34.613467 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.613387 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8a7203a7-e346-47b2-8df1-54f9d2e64d85-proc\") pod \"perf-node-gather-daemonset-r5tst\" (UID: \"8a7203a7-e346-47b2-8df1-54f9d2e64d85\") " pod="openshift-must-gather-qzjtz/perf-node-gather-daemonset-r5tst" Jul 06 12:11:34.714257 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.714170 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkhtn\" (UniqueName: \"kubernetes.io/projected/8a7203a7-e346-47b2-8df1-54f9d2e64d85-kube-api-access-gkhtn\") pod \"perf-node-gather-daemonset-r5tst\" (UID: \"8a7203a7-e346-47b2-8df1-54f9d2e64d85\") " pod="openshift-must-gather-qzjtz/perf-node-gather-daemonset-r5tst" Jul 06 12:11:34.714257 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.714205 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8a7203a7-e346-47b2-8df1-54f9d2e64d85-podres\") pod \"perf-node-gather-daemonset-r5tst\" (UID: \"8a7203a7-e346-47b2-8df1-54f9d2e64d85\") " pod="openshift-must-gather-qzjtz/perf-node-gather-daemonset-r5tst" Jul 06 12:11:34.714257 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.714239 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a7203a7-e346-47b2-8df1-54f9d2e64d85-lib-modules\") pod \"perf-node-gather-daemonset-r5tst\" (UID: \"8a7203a7-e346-47b2-8df1-54f9d2e64d85\") " pod="openshift-must-gather-qzjtz/perf-node-gather-daemonset-r5tst" Jul 06 12:11:34.714547 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.714273 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a7203a7-e346-47b2-8df1-54f9d2e64d85-sys\") pod \"perf-node-gather-daemonset-r5tst\" (UID: \"8a7203a7-e346-47b2-8df1-54f9d2e64d85\") " pod="openshift-must-gather-qzjtz/perf-node-gather-daemonset-r5tst" Jul 06 12:11:34.714547 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.714289 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8a7203a7-e346-47b2-8df1-54f9d2e64d85-proc\") pod \"perf-node-gather-daemonset-r5tst\" (UID: \"8a7203a7-e346-47b2-8df1-54f9d2e64d85\") " pod="openshift-must-gather-qzjtz/perf-node-gather-daemonset-r5tst" Jul 06 12:11:34.714547 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.714370 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8a7203a7-e346-47b2-8df1-54f9d2e64d85-proc\") pod \"perf-node-gather-daemonset-r5tst\" (UID: \"8a7203a7-e346-47b2-8df1-54f9d2e64d85\") " pod="openshift-must-gather-qzjtz/perf-node-gather-daemonset-r5tst" Jul 06 12:11:34.714547 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.714362 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8a7203a7-e346-47b2-8df1-54f9d2e64d85-podres\") pod \"perf-node-gather-daemonset-r5tst\" (UID: \"8a7203a7-e346-47b2-8df1-54f9d2e64d85\") " pod="openshift-must-gather-qzjtz/perf-node-gather-daemonset-r5tst" Jul 06 12:11:34.714547 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.714386 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a7203a7-e346-47b2-8df1-54f9d2e64d85-sys\") pod \"perf-node-gather-daemonset-r5tst\" (UID: \"8a7203a7-e346-47b2-8df1-54f9d2e64d85\") " pod="openshift-must-gather-qzjtz/perf-node-gather-daemonset-r5tst" Jul 06 12:11:34.714547 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.714423 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a7203a7-e346-47b2-8df1-54f9d2e64d85-lib-modules\") pod \"perf-node-gather-daemonset-r5tst\" (UID: \"8a7203a7-e346-47b2-8df1-54f9d2e64d85\") " pod="openshift-must-gather-qzjtz/perf-node-gather-daemonset-r5tst" Jul 06 12:11:34.721760 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.721737 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkhtn\" (UniqueName: \"kubernetes.io/projected/8a7203a7-e346-47b2-8df1-54f9d2e64d85-kube-api-access-gkhtn\") pod \"perf-node-gather-daemonset-r5tst\" (UID: \"8a7203a7-e346-47b2-8df1-54f9d2e64d85\") " pod="openshift-must-gather-qzjtz/perf-node-gather-daemonset-r5tst" Jul 06 12:11:34.725286 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.725258 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qzjtz/perf-node-gather-daemonset-r5tst" Jul 06 12:11:34.857314 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:34.857286 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qzjtz/perf-node-gather-daemonset-r5tst"] Jul 06 12:11:34.858557 ip-10-0-131-176 kubenswrapper[2568]: W0706 12:11:34.858527 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8a7203a7_e346_47b2_8df1_54f9d2e64d85.slice/crio-b9b5b1cacc750d7911bd8ebc1b2f9123da94eb079b82cf4dfdbdcc5c18527ab2 WatchSource:0}: Error finding container b9b5b1cacc750d7911bd8ebc1b2f9123da94eb079b82cf4dfdbdcc5c18527ab2: Status 404 returned error can't find the container with id b9b5b1cacc750d7911bd8ebc1b2f9123da94eb079b82cf4dfdbdcc5c18527ab2 Jul 06 12:11:35.500085 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:35.500043 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qzjtz/perf-node-gather-daemonset-r5tst" event={"ID":"8a7203a7-e346-47b2-8df1-54f9d2e64d85","Type":"ContainerStarted","Data":"78c8b9cfdb71f551fb64adeb7e6e5985f4d4aba25acf427bcb6b4fac36ccba45"} Jul 06 12:11:35.500516 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:35.500208 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qzjtz/perf-node-gather-daemonset-r5tst" Jul 06 12:11:35.500516 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:35.500252 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qzjtz/perf-node-gather-daemonset-r5tst" event={"ID":"8a7203a7-e346-47b2-8df1-54f9d2e64d85","Type":"ContainerStarted","Data":"b9b5b1cacc750d7911bd8ebc1b2f9123da94eb079b82cf4dfdbdcc5c18527ab2"} Jul 06 12:11:35.519048 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:35.518994 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qzjtz/perf-node-gather-daemonset-r5tst" podStartSLOduration=1.518980539 podStartE2EDuration="1.518980539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 12:11:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 12:11:35.516532228 +0000 UTC m=+2593.996331244" watchObservedRunningTime="2026-07-06 12:11:35.518980539 +0000 UTC m=+2593.998779555" Jul 06 12:11:37.548387 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:37.548353 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dvbz4_03ee7031-c88a-47a2-b37c-0349a1787567/dns/0.log" Jul 06 12:11:37.568754 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:37.568710 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dvbz4_03ee7031-c88a-47a2-b37c-0349a1787567/kube-rbac-proxy/0.log" Jul 06 12:11:37.673127 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:37.673102 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dw874_10439038-5c07-452f-bea0-a01a60ef437f/dns-node-resolver/0.log" Jul 06 12:11:38.328605 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:38.328574 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pbv8c_733213c7-49ab-4f45-a0dd-d87abe31ef28/node-ca/0.log" Jul 06 12:11:39.278597 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:39.278567 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678n6qkn_b9859ff5-1763-4c0c-a074-3e9f86a65f68/istio-proxy/0.log" Jul 06 12:11:39.466587 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:39.466553 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-56gcm_e2c53d85-1c9b-4fba-a7b1-05cff745478b/discovery/0.log" Jul 06 12:11:39.491313 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:39.491287 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64c57c544d-4nq7x_a7dbffb2-b3ec-4aea-a451-ef4ad10ab5f4/kube-auth-proxy/0.log" Jul 06 12:11:40.434984 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:40.434951 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-khjfx_e130dede-7e5f-4cd5-8a39-d47ad7f48a2c/serve-healthcheck-canary/0.log" Jul 06 12:11:40.920496 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:40.920467 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-gsnkd_8a12983e-eac8-4b8b-8baf-688b9f7c4907/insights-operator/0.log" Jul 06 12:11:40.921201 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:40.921177 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-gsnkd_8a12983e-eac8-4b8b-8baf-688b9f7c4907/insights-operator/1.log" Jul 06 12:11:41.014059 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:41.014025 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tgp62_eddc93f7-ea09-405e-9488-7c7e5f710ac2/kube-rbac-proxy/0.log" Jul 06 12:11:41.034009 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:41.033987 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tgp62_eddc93f7-ea09-405e-9488-7c7e5f710ac2/exporter/0.log" Jul 06 12:11:41.058449 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:41.058432 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tgp62_eddc93f7-ea09-405e-9488-7c7e5f710ac2/extractor/0.log" Jul 06 12:11:41.515775 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:41.515750 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qzjtz/perf-node-gather-daemonset-r5tst" Jul 06 12:11:43.187647 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:43.187614 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-gh8vj_609ac905-0773-4403-8e9b-8fe8a4dcf908/manager/0.log" Jul 06 12:11:43.400382 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:43.400349 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-6zprf_4db87dba-c23e-4a77-b58b-b9de0644d486/manager/0.log" Jul 06 12:11:43.448651 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:43.448572 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-ccdfff8b-g5tjf_16d80f49-fc68-452e-ab42-eb4dd72a1dfe/maas-api/0.log" Jul 06 12:11:43.472500 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:43.472473 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722305-dwsj4_a5604901-4af8-41b7-be0e-92cc18b812b1/cleanup/0.log" Jul 06 12:11:43.498348 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:43.498327 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722320-rlln9_bf2357b0-6395-4bb3-8bfd-6099d105a418/cleanup/0.log" Jul 06 12:11:43.654898 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:43.654868 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5fc875b447-dlqtv_27876b93-9b29-42c7-b21e-902222ff3daa/manager/0.log" Jul 06 12:11:43.678397 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:43.678365 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-95wlx_dfcbe27d-5c42-45f9-a719-5b43b710c11d/server/0.log" Jul 06 12:11:43.783650 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:43.783572 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-kdsrp_6bfe9a65-bdfc-431b-b376-40f507daf37a/manager/0.log" Jul 06 12:11:43.952299 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:43.952275 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cb7946974-kcmtw_ce314f78-7470-45c3-9a73-cede06d44b62/manager/0.log" Jul 06 12:11:43.969573 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:43.969548 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cb7946974-kcmtw_ce314f78-7470-45c3-9a73-cede06d44b62/copy-manifests/0.log" Jul 06 12:11:44.269562 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:44.269488 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678n6qkn_b9859ff5-1763-4c0c-a074-3e9f86a65f68/istio-proxy/0.log" Jul 06 12:11:44.473560 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:44.473526 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-56gcm_e2c53d85-1c9b-4fba-a7b1-05cff745478b/discovery/0.log" Jul 06 12:11:44.487574 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:44.487551 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64c57c544d-4nq7x_a7dbffb2-b3ec-4aea-a451-ef4ad10ab5f4/kube-auth-proxy/0.log" Jul 06 12:11:45.588848 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:45.588816 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-lwb7h_1be8510e-b7b9-4c5b-93ba-5c14cb7392e0/openshift-lws-operator/0.log" Jul 06 12:11:46.097579 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:46.097550 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678n6qkn_b9859ff5-1763-4c0c-a074-3e9f86a65f68/istio-proxy/0.log" Jul 06 12:11:46.280379 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:46.280351 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-56gcm_e2c53d85-1c9b-4fba-a7b1-05cff745478b/discovery/0.log" Jul 06 12:11:46.294440 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:46.294411 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64c57c544d-4nq7x_a7dbffb2-b3ec-4aea-a451-ef4ad10ab5f4/kube-auth-proxy/0.log" Jul 06 12:11:50.743948 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:50.743910 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-5cq2k_2864a14b-0f18-45e1-8c00-129a0c26570c/kube-storage-version-migrator-operator/1.log" Jul 06 12:11:50.745397 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:50.745369 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-5cq2k_2864a14b-0f18-45e1-8c00-129a0c26570c/kube-storage-version-migrator-operator/0.log" Jul 06 12:11:51.809458 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:51.809428 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-25hzg_cfcd9856-b786-4cb9-b407-1fe4eafe748e/kube-multus-additional-cni-plugins/0.log" Jul 06 12:11:51.831367 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:51.831338 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-25hzg_cfcd9856-b786-4cb9-b407-1fe4eafe748e/egress-router-binary-copy/0.log" Jul 06 12:11:51.854253 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:51.854227 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-25hzg_cfcd9856-b786-4cb9-b407-1fe4eafe748e/cni-plugins/0.log" Jul 06 12:11:51.876453 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:51.876427 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-25hzg_cfcd9856-b786-4cb9-b407-1fe4eafe748e/bond-cni-plugin/0.log" Jul 06 12:11:51.896695 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:51.896653 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-25hzg_cfcd9856-b786-4cb9-b407-1fe4eafe748e/routeoverride-cni/0.log" Jul 06 12:11:51.933986 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:51.933963 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-25hzg_cfcd9856-b786-4cb9-b407-1fe4eafe748e/whereabouts-cni-bincopy/0.log" Jul 06 12:11:51.962596 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:51.962538 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-25hzg_cfcd9856-b786-4cb9-b407-1fe4eafe748e/whereabouts-cni/0.log" Jul 06 12:11:52.389229 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:52.389194 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-pvz5d_e3cd243e-9aaa-4671-b9b8-2f135c474488/kube-multus/0.log" Jul 06 12:11:52.493057 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:52.493021 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rgmk8_6730cd7d-c083-452b-b123-e2e9f92745a1/network-metrics-daemon/0.log" Jul 06 12:11:52.510129 ip-10-0-131-176 kubenswrapper[2568]: I0706 12:11:52.510100 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rgmk8_6730cd7d-c083-452b-b123-e2e9f92745a1/kube-rbac-proxy/0.log"