Jul 18 00:40:39.537978 ip-10-0-134-239 systemd[1]: Starting Kubernetes Kubelet... Jul 18 00:40:39.900902 ip-10-0-134-239 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 18 00:40:39.900902 ip-10-0-134-239 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 18 00:40:39.900902 ip-10-0-134-239 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 18 00:40:39.900902 ip-10-0-134-239 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 18 00:40:39.900902 ip-10-0-134-239 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 18 00:40:39.901619 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.900966 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 18 00:40:39.905634 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905614 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 00:40:39.905634 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905633 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 00:40:39.905721 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905637 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 00:40:39.905721 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905641 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 00:40:39.905721 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905644 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 00:40:39.905721 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905647 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 00:40:39.905721 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905649 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 00:40:39.905721 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905652 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 00:40:39.905721 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905655 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 00:40:39.905721 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905657 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 00:40:39.905721 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905659 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 00:40:39.905721 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905662 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 00:40:39.905721 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905680 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 00:40:39.905721 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905685 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 00:40:39.905721 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905688 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 00:40:39.905721 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905692 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 00:40:39.905721 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905701 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 00:40:39.905721 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905706 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 00:40:39.905721 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905710 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 00:40:39.905721 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905712 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 00:40:39.905721 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905728 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 00:40:39.906166 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905732 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 00:40:39.906166 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905734 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 00:40:39.906166 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905737 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 00:40:39.906166 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905740 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 00:40:39.906166 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905742 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 00:40:39.906166 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905745 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 00:40:39.906166 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905748 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 00:40:39.906166 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905750 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 00:40:39.906166 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905753 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 00:40:39.906166 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905755 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 00:40:39.906166 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905758 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 00:40:39.906166 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905760 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 00:40:39.906166 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905762 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 00:40:39.906166 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905765 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 00:40:39.906166 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905768 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 00:40:39.906166 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905770 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 00:40:39.906166 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905772 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 00:40:39.906166 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905776 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 00:40:39.906166 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905778 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 00:40:39.906166 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905780 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 00:40:39.906642 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905783 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 00:40:39.906642 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905785 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 00:40:39.906642 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905787 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 00:40:39.906642 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905790 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 00:40:39.906642 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905792 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 00:40:39.906642 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905795 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 00:40:39.906642 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905798 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 00:40:39.906642 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905801 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 00:40:39.906642 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905804 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 18 00:40:39.906642 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905807 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 00:40:39.906642 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905809 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 00:40:39.906642 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905812 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 00:40:39.906642 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905815 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 00:40:39.906642 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905818 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 00:40:39.906642 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905820 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 00:40:39.906642 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905823 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 00:40:39.906642 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905825 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 00:40:39.906642 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905828 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 00:40:39.906642 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905830 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 00:40:39.906642 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905832 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 00:40:39.907139 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905835 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 00:40:39.907139 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905837 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 00:40:39.907139 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905839 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 00:40:39.907139 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905842 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 00:40:39.907139 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905844 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 00:40:39.907139 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905846 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 00:40:39.907139 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905849 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 00:40:39.907139 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905852 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 00:40:39.907139 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905854 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 00:40:39.907139 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905856 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 00:40:39.907139 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905859 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 00:40:39.907139 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905862 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 00:40:39.907139 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905865 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 00:40:39.907139 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905867 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 00:40:39.907139 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905870 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 00:40:39.907139 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905872 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 00:40:39.907139 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905875 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 00:40:39.907139 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905877 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 00:40:39.907139 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905880 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 00:40:39.907139 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905882 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 00:40:39.907611 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905884 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 00:40:39.907611 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905888 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 00:40:39.907611 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905890 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 00:40:39.907611 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905893 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 00:40:39.907611 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905895 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 00:40:39.907611 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905898 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 00:40:39.907611 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905901 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 00:40:39.907611 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905905 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 00:40:39.907611 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905909 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 00:40:39.907611 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905912 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 00:40:39.907611 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905914 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 00:40:39.907611 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905917 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 00:40:39.907611 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905919 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 00:40:39.907611 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905923 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 00:40:39.907611 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905925 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 00:40:39.907611 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905928 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 00:40:39.907611 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905931 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 00:40:39.907611 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.905933 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 00:40:39.907611 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906691 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 00:40:39.908086 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906710 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 00:40:39.908086 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906713 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 00:40:39.908086 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906716 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 00:40:39.908086 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906718 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 00:40:39.908086 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906721 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 00:40:39.908086 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906724 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 00:40:39.908086 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906726 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 00:40:39.908086 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906730 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 00:40:39.908086 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906732 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 00:40:39.908086 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906734 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 00:40:39.908086 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906737 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 00:40:39.908086 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906739 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 00:40:39.908086 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906741 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 00:40:39.908086 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906744 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 00:40:39.908086 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906747 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 00:40:39.908086 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906751 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 00:40:39.908086 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906754 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 00:40:39.908086 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906756 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 00:40:39.908086 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906759 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 00:40:39.908086 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906768 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 00:40:39.908577 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906772 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 00:40:39.908577 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906774 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 00:40:39.908577 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906777 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 00:40:39.908577 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906781 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 00:40:39.908577 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906784 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 00:40:39.908577 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906787 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 00:40:39.908577 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906789 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 00:40:39.908577 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906792 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 00:40:39.908577 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906794 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 00:40:39.908577 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906799 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 00:40:39.908577 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906803 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 00:40:39.908577 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906806 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 00:40:39.908577 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906808 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 00:40:39.908577 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906811 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 00:40:39.908577 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906813 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 00:40:39.908577 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906816 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 00:40:39.908577 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906818 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 00:40:39.908577 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906821 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 00:40:39.908577 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906823 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 00:40:39.909051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906826 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 00:40:39.909051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906828 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 00:40:39.909051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906831 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 00:40:39.909051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906833 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 00:40:39.909051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906836 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 00:40:39.909051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906838 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 00:40:39.909051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906841 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 00:40:39.909051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906843 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 00:40:39.909051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906846 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 00:40:39.909051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906849 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 00:40:39.909051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906851 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 00:40:39.909051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906854 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 00:40:39.909051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906857 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 00:40:39.909051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906869 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 00:40:39.909051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906871 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 00:40:39.909051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906874 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 00:40:39.909051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906876 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 00:40:39.909051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906879 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 00:40:39.909051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906881 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 00:40:39.909519 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906884 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 00:40:39.909519 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906886 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 00:40:39.909519 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906889 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 00:40:39.909519 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906891 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 00:40:39.909519 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906894 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 00:40:39.909519 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906896 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 00:40:39.909519 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906899 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 00:40:39.909519 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906901 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 00:40:39.909519 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906903 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 00:40:39.909519 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906907 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 00:40:39.909519 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906909 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 18 00:40:39.909519 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906911 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 00:40:39.909519 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906914 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 00:40:39.909519 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906917 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 00:40:39.909519 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906919 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 00:40:39.909519 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906921 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 00:40:39.909519 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906923 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 00:40:39.909519 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906926 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 00:40:39.909519 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906928 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 00:40:39.909519 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906931 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 00:40:39.910112 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906933 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 00:40:39.910112 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906935 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 00:40:39.910112 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906938 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 00:40:39.910112 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906942 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 00:40:39.910112 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906946 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 00:40:39.910112 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906949 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 00:40:39.910112 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906952 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 00:40:39.910112 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906956 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 00:40:39.910112 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906958 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 00:40:39.910112 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906961 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 00:40:39.910112 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906964 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 00:40:39.910112 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906966 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 00:40:39.910112 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906970 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 00:40:39.910112 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906974 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 00:40:39.910112 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906977 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 00:40:39.910112 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906980 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 00:40:39.910112 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906983 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 00:40:39.910112 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906986 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 00:40:39.910112 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906988 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 00:40:39.910567 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.906991 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 00:40:39.910567 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907399 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 00:40:39.910567 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907403 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 00:40:39.910567 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907407 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 00:40:39.910567 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907409 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 00:40:39.910567 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907412 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 00:40:39.910567 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907415 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 00:40:39.910567 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907417 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 00:40:39.910567 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907419 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 00:40:39.910567 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907422 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 00:40:39.910567 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907424 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 00:40:39.910567 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907426 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 00:40:39.910567 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907429 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 00:40:39.910567 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907431 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 00:40:39.910567 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907433 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 00:40:39.910567 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907436 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 00:40:39.910567 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907438 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 18 00:40:39.910567 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907441 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 00:40:39.910567 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907444 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 00:40:39.910567 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907446 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 00:40:39.911063 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907449 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 00:40:39.911063 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907452 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 00:40:39.911063 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907454 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 00:40:39.911063 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907457 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 00:40:39.911063 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907459 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 00:40:39.911063 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907462 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 00:40:39.911063 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907464 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 00:40:39.911063 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907467 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 00:40:39.911063 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907471 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 00:40:39.911063 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907475 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 00:40:39.911063 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907478 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 00:40:39.911063 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907480 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 00:40:39.911063 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907483 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 00:40:39.911063 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907486 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 00:40:39.911063 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907488 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 00:40:39.911063 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907491 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 00:40:39.911063 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907493 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 00:40:39.911063 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907496 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 00:40:39.911063 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907499 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 00:40:39.911063 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907501 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 00:40:39.911563 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907504 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 00:40:39.911563 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907506 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 00:40:39.911563 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907508 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 00:40:39.911563 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907511 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 00:40:39.911563 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907513 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 00:40:39.911563 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907515 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 00:40:39.911563 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907518 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 00:40:39.911563 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907520 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 00:40:39.911563 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907522 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 00:40:39.911563 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907525 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 00:40:39.911563 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907528 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 00:40:39.911563 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907530 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 00:40:39.911563 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907532 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 00:40:39.911563 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907535 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 00:40:39.911563 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907538 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 00:40:39.911563 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907540 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 00:40:39.911563 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907543 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 00:40:39.911563 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907546 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 00:40:39.911563 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907548 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 00:40:39.912029 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907550 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 00:40:39.912029 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907552 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 00:40:39.912029 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907555 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 00:40:39.912029 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907557 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 00:40:39.912029 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907560 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 00:40:39.912029 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907562 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 00:40:39.912029 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907564 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 00:40:39.912029 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907567 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 00:40:39.912029 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907569 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 00:40:39.912029 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907572 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 00:40:39.912029 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907576 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 00:40:39.912029 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907580 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 00:40:39.912029 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907584 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 00:40:39.912029 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907587 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 00:40:39.912029 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907590 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 00:40:39.912029 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907593 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 00:40:39.912029 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907596 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 00:40:39.912029 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907598 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 00:40:39.912029 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907601 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 00:40:39.912029 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907604 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 00:40:39.912501 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907606 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 00:40:39.912501 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907609 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 00:40:39.912501 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907611 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 00:40:39.912501 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907614 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 00:40:39.912501 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907617 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 00:40:39.912501 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907619 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 00:40:39.912501 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907622 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 00:40:39.912501 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907627 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 00:40:39.912501 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907630 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 00:40:39.912501 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907633 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 00:40:39.912501 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907635 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 00:40:39.912501 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907637 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 00:40:39.912501 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907640 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 00:40:39.912501 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907642 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 00:40:39.912501 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907644 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 00:40:39.912501 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907647 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 00:40:39.912501 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907649 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 00:40:39.912501 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907652 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 00:40:39.912501 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907655 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 00:40:39.912997 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907658 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 00:40:39.912997 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.907661 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 00:40:39.912997 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907749 2580 flags.go:64] FLAG: --address="0.0.0.0" Jul 18 00:40:39.912997 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907757 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 18 00:40:39.912997 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907764 2580 flags.go:64] FLAG: --anonymous-auth="true" Jul 18 00:40:39.912997 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907769 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 18 00:40:39.912997 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907773 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 18 00:40:39.912997 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907777 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 18 00:40:39.912997 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907781 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 18 00:40:39.912997 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907785 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 18 00:40:39.912997 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907788 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 18 00:40:39.912997 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907791 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 18 00:40:39.912997 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907795 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 18 00:40:39.912997 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907798 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 18 00:40:39.912997 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907801 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 18 00:40:39.912997 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907804 2580 flags.go:64] FLAG: --cgroup-root="" Jul 18 00:40:39.912997 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907807 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 18 00:40:39.912997 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907810 2580 flags.go:64] FLAG: --client-ca-file="" Jul 18 00:40:39.912997 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907813 2580 flags.go:64] FLAG: --cloud-provider="external" Jul 18 00:40:39.912997 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907815 2580 flags.go:64] FLAG: --cluster-dns="[]" Jul 18 00:40:39.912997 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907820 2580 flags.go:64] FLAG: --cluster-domain="" Jul 18 00:40:39.912997 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907823 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 18 00:40:39.912997 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907826 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 18 00:40:39.913555 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907829 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 18 00:40:39.913555 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907832 2580 flags.go:64] FLAG: --container-log-max-files="5" Jul 18 00:40:39.913555 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907836 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 18 00:40:39.913555 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907839 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 18 00:40:39.913555 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907843 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 18 00:40:39.913555 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907846 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 18 00:40:39.913555 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907848 2580 flags.go:64] FLAG: --contention-profiling="false" Jul 18 00:40:39.913555 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907851 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 18 00:40:39.913555 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907854 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 18 00:40:39.913555 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907857 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 18 00:40:39.913555 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907860 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 18 00:40:39.913555 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907864 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 18 00:40:39.913555 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907869 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 18 00:40:39.913555 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907872 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 18 00:40:39.913555 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907875 2580 flags.go:64] FLAG: --enable-load-reader="false" Jul 18 00:40:39.913555 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907878 2580 flags.go:64] FLAG: --enable-server="true" Jul 18 00:40:39.913555 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907881 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 18 00:40:39.913555 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907885 2580 flags.go:64] FLAG: --event-burst="100" Jul 18 00:40:39.913555 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907888 2580 flags.go:64] FLAG: --event-qps="50" Jul 18 00:40:39.913555 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907891 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 18 00:40:39.913555 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907894 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 18 00:40:39.913555 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907897 2580 flags.go:64] FLAG: --eviction-hard="" Jul 18 00:40:39.913555 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907901 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 18 00:40:39.913555 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907903 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 18 00:40:39.914181 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907906 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 18 00:40:39.914181 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907910 2580 flags.go:64] FLAG: --eviction-soft="" Jul 18 00:40:39.914181 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907912 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 18 00:40:39.914181 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907915 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 18 00:40:39.914181 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907918 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 18 00:40:39.914181 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907921 2580 flags.go:64] FLAG: --experimental-mounter-path="" Jul 18 00:40:39.914181 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907924 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 18 00:40:39.914181 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907928 2580 flags.go:64] FLAG: --fail-swap-on="true" Jul 18 00:40:39.914181 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907930 2580 flags.go:64] FLAG: --feature-gates="" Jul 18 00:40:39.914181 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907934 2580 flags.go:64] FLAG: --file-check-frequency="20s" Jul 18 00:40:39.914181 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907937 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 18 00:40:39.914181 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907940 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 18 00:40:39.914181 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907943 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 18 00:40:39.914181 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907946 2580 flags.go:64] FLAG: --healthz-port="10248" Jul 18 00:40:39.914181 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907949 2580 flags.go:64] FLAG: --help="false" Jul 18 00:40:39.914181 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907952 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-134-239.ec2.internal" Jul 18 00:40:39.914181 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907955 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 18 00:40:39.914181 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907958 2580 flags.go:64] FLAG: --http-check-frequency="20s" Jul 18 00:40:39.914181 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907961 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 18 00:40:39.914181 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907964 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 18 00:40:39.914181 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907969 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 18 00:40:39.914181 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907972 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 18 00:40:39.914181 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907975 2580 flags.go:64] FLAG: --image-service-endpoint="" Jul 18 00:40:39.914957 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907977 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 18 00:40:39.914957 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907980 2580 flags.go:64] FLAG: --kube-api-burst="100" Jul 18 00:40:39.914957 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907983 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 18 00:40:39.914957 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907986 2580 flags.go:64] FLAG: --kube-api-qps="50" Jul 18 00:40:39.914957 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907989 2580 flags.go:64] FLAG: --kube-reserved="" Jul 18 00:40:39.914957 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907992 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 18 00:40:39.914957 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907994 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 18 00:40:39.914957 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.907997 2580 flags.go:64] FLAG: --kubelet-cgroups="" Jul 18 00:40:39.914957 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908000 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 18 00:40:39.914957 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908003 2580 flags.go:64] FLAG: --lock-file="" Jul 18 00:40:39.914957 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908006 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 18 00:40:39.914957 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908008 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 18 00:40:39.914957 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908011 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 18 00:40:39.914957 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908017 2580 flags.go:64] FLAG: --log-json-split-stream="false" Jul 18 00:40:39.914957 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908019 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 18 00:40:39.914957 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908022 2580 flags.go:64] FLAG: --log-text-split-stream="false" Jul 18 00:40:39.914957 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908025 2580 flags.go:64] FLAG: --logging-format="text" Jul 18 00:40:39.914957 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908029 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 18 00:40:39.914957 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908032 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 18 00:40:39.914957 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908035 2580 flags.go:64] FLAG: --manifest-url="" Jul 18 00:40:39.914957 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908037 2580 flags.go:64] FLAG: --manifest-url-header="" Jul 18 00:40:39.914957 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908042 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 18 00:40:39.914957 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908045 2580 flags.go:64] FLAG: --max-open-files="1000000" Jul 18 00:40:39.914957 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908049 2580 flags.go:64] FLAG: --max-pods="110" Jul 18 00:40:39.914957 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908051 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 18 00:40:39.915776 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908054 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 18 00:40:39.915776 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908058 2580 flags.go:64] FLAG: --memory-manager-policy="None" Jul 18 00:40:39.915776 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908061 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 18 00:40:39.915776 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908064 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 18 00:40:39.915776 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908070 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 18 00:40:39.915776 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908073 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 18 00:40:39.915776 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908080 2580 flags.go:64] FLAG: --node-status-max-images="50" Jul 18 00:40:39.915776 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908083 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 18 00:40:39.915776 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908085 2580 flags.go:64] FLAG: --oom-score-adj="-999" Jul 18 00:40:39.915776 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908088 2580 flags.go:64] FLAG: --pod-cidr="" Jul 18 00:40:39.915776 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908091 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 18 00:40:39.915776 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908097 2580 flags.go:64] FLAG: --pod-manifest-path="" Jul 18 00:40:39.915776 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908100 2580 flags.go:64] FLAG: --pod-max-pids="-1" Jul 18 00:40:39.915776 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908103 2580 flags.go:64] FLAG: --pods-per-core="0" Jul 18 00:40:39.915776 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908106 2580 flags.go:64] FLAG: --port="10250" Jul 18 00:40:39.915776 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908109 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 18 00:40:39.915776 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908112 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0927459fe7fac36a3" Jul 18 00:40:39.915776 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908114 2580 flags.go:64] FLAG: --qos-reserved="" Jul 18 00:40:39.915776 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908117 2580 flags.go:64] FLAG: --read-only-port="10255" Jul 18 00:40:39.915776 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908120 2580 flags.go:64] FLAG: --register-node="true" Jul 18 00:40:39.915776 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908123 2580 flags.go:64] FLAG: --register-with-taints="" Jul 18 00:40:39.915776 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908126 2580 flags.go:64] FLAG: --registry-burst="10" Jul 18 00:40:39.915776 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908129 2580 flags.go:64] FLAG: --registry-qps="5" Jul 18 00:40:39.915776 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908131 2580 flags.go:64] FLAG: --reserved-cpus="" Jul 18 00:40:39.916367 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908134 2580 flags.go:64] FLAG: --reserved-memory="" Jul 18 00:40:39.916367 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908138 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 18 00:40:39.916367 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908141 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 18 00:40:39.916367 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908144 2580 flags.go:64] FLAG: --rotate-certificates="false" Jul 18 00:40:39.916367 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908147 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 18 00:40:39.916367 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908149 2580 flags.go:64] FLAG: --runonce="false" Jul 18 00:40:39.916367 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908152 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 18 00:40:39.916367 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908155 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 18 00:40:39.916367 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908158 2580 flags.go:64] FLAG: --seccomp-default="false" Jul 18 00:40:39.916367 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908160 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 18 00:40:39.916367 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908164 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 18 00:40:39.916367 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908167 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 18 00:40:39.916367 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908171 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 18 00:40:39.916367 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908174 2580 flags.go:64] FLAG: --storage-driver-password="root" Jul 18 00:40:39.916367 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908177 2580 flags.go:64] FLAG: --storage-driver-secure="false" Jul 18 00:40:39.916367 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908180 2580 flags.go:64] FLAG: --storage-driver-table="stats" Jul 18 00:40:39.916367 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908182 2580 flags.go:64] FLAG: --storage-driver-user="root" Jul 18 00:40:39.916367 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908185 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 18 00:40:39.916367 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908188 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 18 00:40:39.916367 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908191 2580 flags.go:64] FLAG: --system-cgroups="" Jul 18 00:40:39.916367 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908194 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 18 00:40:39.916367 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908199 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 18 00:40:39.916367 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908202 2580 flags.go:64] FLAG: --tls-cert-file="" Jul 18 00:40:39.916367 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908205 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 18 00:40:39.916367 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908209 2580 flags.go:64] FLAG: --tls-min-version="" Jul 18 00:40:39.917040 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908211 2580 flags.go:64] FLAG: --tls-private-key-file="" Jul 18 00:40:39.917040 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908214 2580 flags.go:64] FLAG: --topology-manager-policy="none" Jul 18 00:40:39.917040 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908217 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 18 00:40:39.917040 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908220 2580 flags.go:64] FLAG: --topology-manager-scope="container" Jul 18 00:40:39.917040 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908222 2580 flags.go:64] FLAG: --v="2" Jul 18 00:40:39.917040 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908226 2580 flags.go:64] FLAG: --version="false" Jul 18 00:40:39.917040 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908230 2580 flags.go:64] FLAG: --vmodule="" Jul 18 00:40:39.917040 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908234 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 18 00:40:39.917040 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908237 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 18 00:40:39.917040 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908353 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 00:40:39.917040 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908356 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 00:40:39.917040 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908360 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 00:40:39.917040 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908363 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 00:40:39.917040 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908365 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 00:40:39.917040 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908368 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 00:40:39.917040 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908371 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 00:40:39.917040 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908373 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 00:40:39.917040 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908377 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 00:40:39.917040 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908380 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 00:40:39.917040 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908384 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 00:40:39.917040 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908386 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 00:40:39.917569 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908389 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 00:40:39.917569 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908392 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 00:40:39.917569 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908394 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 00:40:39.917569 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908397 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 00:40:39.917569 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908399 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 00:40:39.917569 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908402 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 00:40:39.917569 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908404 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 00:40:39.917569 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908407 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 00:40:39.917569 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908409 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 00:40:39.917569 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908412 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 00:40:39.917569 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908416 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 00:40:39.917569 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908419 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 00:40:39.917569 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908422 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 18 00:40:39.917569 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908426 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 00:40:39.917569 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908430 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 00:40:39.917569 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908433 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 00:40:39.917569 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908435 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 00:40:39.917569 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908438 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 00:40:39.917569 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908440 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 00:40:39.918051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908444 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 00:40:39.918051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908447 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 00:40:39.918051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908451 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 00:40:39.918051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908453 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 00:40:39.918051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908456 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 00:40:39.918051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908459 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 00:40:39.918051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908462 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 00:40:39.918051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908465 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 00:40:39.918051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908468 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 00:40:39.918051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908471 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 00:40:39.918051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908474 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 00:40:39.918051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908478 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 00:40:39.918051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908481 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 00:40:39.918051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908483 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 00:40:39.918051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908486 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 00:40:39.918051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908488 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 00:40:39.918051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908491 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 00:40:39.918051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908493 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 00:40:39.918051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908496 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 00:40:39.918051 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908498 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 00:40:39.918558 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908501 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 00:40:39.918558 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908504 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 00:40:39.918558 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908506 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 00:40:39.918558 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908508 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 00:40:39.918558 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908511 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 00:40:39.918558 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908513 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 00:40:39.918558 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908516 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 00:40:39.918558 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908518 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 00:40:39.918558 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908521 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 00:40:39.918558 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908524 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 00:40:39.918558 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908526 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 00:40:39.918558 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908529 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 00:40:39.918558 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908531 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 00:40:39.918558 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908534 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 00:40:39.918558 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908537 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 00:40:39.918558 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908540 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 00:40:39.918558 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908542 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 00:40:39.918558 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908545 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 00:40:39.918558 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908547 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 00:40:39.918558 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908550 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 00:40:39.919055 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908552 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 00:40:39.919055 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908556 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 00:40:39.919055 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908559 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 00:40:39.919055 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908562 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 00:40:39.919055 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908565 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 00:40:39.919055 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908567 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 00:40:39.919055 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908570 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 00:40:39.919055 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908572 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 00:40:39.919055 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908575 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 00:40:39.919055 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908577 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 00:40:39.919055 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908580 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 00:40:39.919055 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908583 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 00:40:39.919055 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908586 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 00:40:39.919055 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908588 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 00:40:39.919055 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908591 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 00:40:39.919055 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908594 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 00:40:39.919055 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908596 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 00:40:39.919055 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908599 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 00:40:39.919055 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908601 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 00:40:39.919055 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908604 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 00:40:39.919576 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908606 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 00:40:39.919576 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908608 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 00:40:39.919576 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908611 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 00:40:39.919576 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908614 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 00:40:39.919576 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908616 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 00:40:39.919576 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908619 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 00:40:39.919576 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908621 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 00:40:39.919576 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.908624 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 00:40:39.919576 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.908630 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 18 00:40:39.919576 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.916684 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 18 00:40:39.919576 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.916701 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 18 00:40:39.919576 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.916721 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 18 00:40:39.919576 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.916727 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 18 00:40:39.919576 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916786 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 00:40:39.919970 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916792 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 00:40:39.919970 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916796 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 00:40:39.919970 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916800 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 00:40:39.919970 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916803 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 00:40:39.919970 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916806 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 00:40:39.919970 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916808 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 00:40:39.919970 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916811 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 00:40:39.919970 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916814 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 00:40:39.919970 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916816 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 00:40:39.919970 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916819 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 00:40:39.919970 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916821 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 00:40:39.919970 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916824 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 00:40:39.919970 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916826 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 00:40:39.919970 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916829 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 00:40:39.919970 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916832 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 00:40:39.919970 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916834 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 00:40:39.919970 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916837 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 18 00:40:39.919970 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916840 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 00:40:39.919970 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916843 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 00:40:39.919970 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916845 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 00:40:39.920461 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916848 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 00:40:39.920461 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916851 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 00:40:39.920461 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916853 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 00:40:39.920461 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916856 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 00:40:39.920461 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916858 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 00:40:39.920461 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916860 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 00:40:39.920461 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916863 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 00:40:39.920461 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916866 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 00:40:39.920461 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916869 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 00:40:39.920461 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916871 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 00:40:39.920461 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916875 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 00:40:39.920461 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916879 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 00:40:39.920461 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916882 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 00:40:39.920461 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916886 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 00:40:39.920461 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916888 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 00:40:39.920461 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916891 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 00:40:39.920461 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916894 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 00:40:39.920461 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916897 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 00:40:39.920461 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916899 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 00:40:39.921060 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916902 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 00:40:39.921060 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916904 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 00:40:39.921060 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916907 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 00:40:39.921060 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916909 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 00:40:39.921060 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916911 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 00:40:39.921060 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916914 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 00:40:39.921060 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916916 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 00:40:39.921060 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916919 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 00:40:39.921060 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916922 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 00:40:39.921060 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916924 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 00:40:39.921060 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916927 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 00:40:39.921060 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916930 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 00:40:39.921060 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916932 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 00:40:39.921060 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916934 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 00:40:39.921060 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916937 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 00:40:39.921060 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916939 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 00:40:39.921060 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916942 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 00:40:39.921060 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916945 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 00:40:39.921060 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916947 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 00:40:39.921060 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916949 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 00:40:39.921547 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916953 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 00:40:39.921547 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916955 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 00:40:39.921547 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916958 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 00:40:39.921547 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916960 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 00:40:39.921547 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916963 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 00:40:39.921547 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916965 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 00:40:39.921547 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916968 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 00:40:39.921547 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916971 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 00:40:39.921547 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916973 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 00:40:39.921547 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916976 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 00:40:39.921547 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916978 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 00:40:39.921547 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916981 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 00:40:39.921547 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916983 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 00:40:39.921547 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916985 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 00:40:39.921547 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916988 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 00:40:39.921547 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916991 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 00:40:39.921547 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916993 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 00:40:39.921547 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916995 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 00:40:39.921547 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.916998 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 00:40:39.921547 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917001 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 00:40:39.922097 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917004 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 00:40:39.922097 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917007 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 00:40:39.922097 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917009 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 00:40:39.922097 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917012 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 00:40:39.922097 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917015 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 00:40:39.922097 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917017 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 00:40:39.922097 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917020 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 00:40:39.922097 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917022 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 00:40:39.922097 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917025 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 00:40:39.922097 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917029 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 00:40:39.922097 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917032 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 00:40:39.922097 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917035 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 00:40:39.922097 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917038 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 00:40:39.922097 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917042 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 00:40:39.922097 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917044 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 00:40:39.922097 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917047 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 00:40:39.922097 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917050 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 00:40:39.922097 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917053 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 00:40:39.922097 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917056 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 00:40:39.922553 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.917061 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 18 00:40:39.922553 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917196 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 00:40:39.922553 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917202 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 00:40:39.922553 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917205 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 00:40:39.922553 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917208 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 00:40:39.922553 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917210 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 00:40:39.922553 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917213 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 00:40:39.922553 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917216 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 00:40:39.922553 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917218 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 00:40:39.922553 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917221 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 00:40:39.922553 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917224 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 00:40:39.922553 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917226 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 00:40:39.922553 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917229 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 00:40:39.922553 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917232 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 00:40:39.922553 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917234 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 00:40:39.922937 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917237 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 00:40:39.922937 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917239 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 00:40:39.922937 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917241 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 00:40:39.922937 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917244 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 00:40:39.922937 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917246 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 00:40:39.922937 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917249 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 00:40:39.922937 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917251 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 00:40:39.922937 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917254 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 00:40:39.922937 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917256 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 00:40:39.922937 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917258 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 00:40:39.922937 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917261 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 00:40:39.922937 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917264 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 00:40:39.922937 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917266 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 00:40:39.922937 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917269 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 00:40:39.922937 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917272 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 00:40:39.922937 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917274 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 00:40:39.922937 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917277 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 00:40:39.922937 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917279 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 00:40:39.922937 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917282 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 00:40:39.923393 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917284 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 00:40:39.923393 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917287 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 00:40:39.923393 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917289 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 00:40:39.923393 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917292 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 00:40:39.923393 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917294 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 00:40:39.923393 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917297 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 00:40:39.923393 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917299 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 00:40:39.923393 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917302 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 00:40:39.923393 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917304 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 00:40:39.923393 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917307 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 00:40:39.923393 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917309 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 00:40:39.923393 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917312 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 00:40:39.923393 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917316 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 00:40:39.923393 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917318 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 00:40:39.923393 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917321 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 00:40:39.923393 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917323 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 00:40:39.923393 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917326 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 00:40:39.923393 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917342 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 00:40:39.923393 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917346 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 00:40:39.923896 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917349 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 00:40:39.923896 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917351 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 00:40:39.923896 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917354 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 00:40:39.923896 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917357 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 00:40:39.923896 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917359 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 00:40:39.923896 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917362 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 00:40:39.923896 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917364 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 00:40:39.923896 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917368 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 00:40:39.923896 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917372 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 00:40:39.923896 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917374 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 00:40:39.923896 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917377 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 00:40:39.923896 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917380 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 00:40:39.923896 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917382 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 00:40:39.923896 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917385 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 00:40:39.923896 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917387 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 00:40:39.923896 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917390 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 00:40:39.923896 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917394 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 00:40:39.923896 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917397 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 00:40:39.923896 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917400 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 00:40:39.923896 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917402 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 00:40:39.924366 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917405 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 00:40:39.924366 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917408 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 18 00:40:39.924366 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917410 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 00:40:39.924366 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917413 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 00:40:39.924366 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917415 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 00:40:39.924366 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917418 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 00:40:39.924366 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917421 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 00:40:39.924366 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917425 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 00:40:39.924366 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917427 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 00:40:39.924366 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917430 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 00:40:39.924366 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917432 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 00:40:39.924366 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917435 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 00:40:39.924366 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917438 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 00:40:39.924366 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917440 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 00:40:39.924366 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917443 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 00:40:39.924366 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917445 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 00:40:39.924366 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917448 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 00:40:39.924366 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917451 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 00:40:39.924366 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917453 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 00:40:39.924366 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917455 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 00:40:39.924867 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917458 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 00:40:39.924867 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917460 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 00:40:39.924867 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917463 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 00:40:39.924867 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917465 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 00:40:39.924867 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917468 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 00:40:39.924867 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917470 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 00:40:39.924867 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:39.917473 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 00:40:39.924867 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.917477 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 18 00:40:39.924867 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.917603 2580 server.go:962] "Client rotation is on, will bootstrap in background" Jul 18 00:40:39.924867 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.919588 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 18 00:40:39.924867 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.921601 2580 server.go:1019] "Starting client certificate rotation" Jul 18 00:40:39.924867 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.921703 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 18 00:40:39.924867 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.921734 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 18 00:40:39.925179 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.923821 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 18 00:40:39.926281 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.926264 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 18 00:40:39.939654 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.939632 2580 log.go:25] "Validated CRI v1 runtime API" Jul 18 00:40:39.942493 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.942474 2580 log.go:25] "Validated CRI v1 image API" Jul 18 00:40:39.943401 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.943388 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 18 00:40:39.945800 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.945779 2580 fs.go:135] Filesystem UUIDs: map[7046dfe5-6439-45aa-84bc-d5fe4dfb7c3f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 fd3caef2-dfeb-41f4-bdb8-e58ca6dfa1c0:/dev/nvme0n1p3] Jul 18 00:40:39.945879 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.945799 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 18 00:40:39.950552 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.950531 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 18 00:40:39.953035 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.952926 2580 manager.go:216] Machine: {Timestamp:2026-07-18 00:40:39.951574325 +0000 UTC m=+0.303424660 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102585 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec278eb7fb092d46a5d71225931d3909 SystemUUID:ec278eb7-fb09-2d46-a5d7-1225931d3909 BootID:089131ed-efdb-4c5a-bf70-42282aad199e Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:82:51:07:4d:e1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:82:51:07:4d:e1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0a:ef:b8:1f:79:2c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 18 00:40:39.953035 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.953034 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 18 00:40:39.953153 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.953121 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 18 00:40:39.955696 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.955650 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 18 00:40:39.955859 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.955700 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-239.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 18 00:40:39.956309 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.956300 2580 topology_manager.go:138] "Creating topology manager with none policy" Jul 18 00:40:39.956340 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.956312 2580 container_manager_linux.go:309] "Creating device plugin manager" Jul 18 00:40:39.956340 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.956321 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 18 00:40:39.956941 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.956931 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 18 00:40:39.958089 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.958079 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 18 00:40:39.960779 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.960767 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 18 00:40:39.960910 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.960900 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 18 00:40:39.960939 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.960933 2580 kubelet.go:486] "Attempting to sync node with API server" Jul 18 00:40:39.960965 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.960943 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 18 00:40:39.960965 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.960955 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 18 00:40:39.960965 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.960964 2580 kubelet.go:398] "Adding apiserver pod source" Jul 18 00:40:39.961067 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.960973 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 18 00:40:39.961395 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.961383 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 18 00:40:39.961436 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.961405 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 18 00:40:39.963346 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.963329 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 18 00:40:39.964611 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.964598 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 18 00:40:39.964652 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.964629 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 18 00:40:39.965229 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.965219 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 18 00:40:39.965281 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.965236 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 18 00:40:39.965281 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.965242 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 18 00:40:39.965281 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.965247 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 18 00:40:39.965281 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.965255 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 18 00:40:39.965281 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.965260 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 18 00:40:39.965414 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.965298 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 18 00:40:39.965414 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.965306 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 18 00:40:39.965414 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.965312 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 18 00:40:39.965414 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.965317 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 18 00:40:39.965414 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.965326 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 18 00:40:39.965414 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.965342 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 18 00:40:39.966057 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.966047 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 18 00:40:39.966057 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.966056 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 18 00:40:39.969529 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.969513 2580 server.go:1268] "Started kubelet" Jul 18 00:40:39.969636 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.969606 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 18 00:40:39.969717 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.969620 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 18 00:40:39.969717 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.969662 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 18 00:40:39.970420 ip-10-0-134-239 systemd[1]: Started Kubernetes Kubelet. Jul 18 00:40:39.972146 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.972127 2580 server.go:310] "Adding debug handlers to kubelet server" Jul 18 00:40:39.973977 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.973959 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 18 00:40:39.975014 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.974993 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-239.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 18 00:40:39.975207 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:39.975182 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-239.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 18 00:40:39.975254 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:39.975183 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 18 00:40:39.975449 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:39.974688 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-239.ec2.internal.18c33b726a979b63 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-239.ec2.internal,UID:ip-10-0-134-239.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-239.ec2.internal,},FirstTimestamp:2026-07-18 00:40:39.969479523 +0000 UTC m=+0.321329859,LastTimestamp:2026-07-18 00:40:39.969479523 +0000 UTC m=+0.321329859,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-239.ec2.internal,}" Jul 18 00:40:39.975850 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:39.975819 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 18 00:40:39.979184 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.979162 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 18 00:40:39.979184 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.979180 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 18 00:40:39.979641 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.979506 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2x6k7" Jul 18 00:40:39.979641 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:39.979593 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-239.ec2.internal\" not found" Jul 18 00:40:39.979799 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.979781 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 18 00:40:39.979849 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.979802 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 18 00:40:39.980073 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.979943 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 18 00:40:39.980073 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.980049 2580 reconstruct.go:97] "Volume reconstruction finished" Jul 18 00:40:39.980073 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.980056 2580 reconciler.go:29] "Reconciler: start to sync state" Jul 18 00:40:39.980227 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.980131 2580 factory.go:55] Registering systemd factory Jul 18 00:40:39.980227 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.980146 2580 factory.go:223] Registration of the systemd container factory successfully Jul 18 00:40:39.980378 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.980364 2580 factory.go:153] Registering CRI-O factory Jul 18 00:40:39.980440 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.980381 2580 factory.go:223] Registration of the crio container factory successfully Jul 18 00:40:39.980440 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.980432 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 18 00:40:39.980581 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.980456 2580 factory.go:103] Registering Raw factory Jul 18 00:40:39.980581 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.980471 2580 manager.go:1196] Started watching for new ooms in manager Jul 18 00:40:39.981296 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.981280 2580 manager.go:349] Starting recovery of all containers Jul 18 00:40:39.981773 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:39.981745 2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-239.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 18 00:40:39.981970 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:39.981940 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 18 00:40:39.985916 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.985818 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2x6k7" Jul 18 00:40:39.993436 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.993420 2580 manager.go:354] Recovery completed Jul 18 00:40:39.997575 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.997461 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 00:40:39.999476 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.999459 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-239.ec2.internal" event="NodeHasSufficientMemory" Jul 18 00:40:39.999530 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.999495 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-239.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 00:40:39.999530 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:39.999508 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-239.ec2.internal" event="NodeHasSufficientPID" Jul 18 00:40:40.000115 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.000100 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 18 00:40:40.000115 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.000113 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 18 00:40:40.000115 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.000115 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 00:40:40.000248 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.000126 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 18 00:40:40.000801 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.000786 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-239.ec2.internal" event="NodeHasSufficientMemory" Jul 18 00:40:40.000869 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.000813 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-239.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 00:40:40.000869 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.000825 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-239.ec2.internal" event="NodeHasSufficientPID" Jul 18 00:40:40.001147 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.001135 2580 policy_none.go:49] "None policy: Start" Jul 18 00:40:40.001174 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.001152 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 18 00:40:40.001174 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.001162 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 18 00:40:40.002028 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.002018 2580 policy_none.go:47] "Start" Jul 18 00:40:40.041235 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.041218 2580 manager.go:337] "Starting Device Plugin manager" Jul 18 00:40:40.041346 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:40.041251 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 18 00:40:40.041346 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.041259 2580 server.go:87] "Starting device plugin registration server" Jul 18 00:40:40.041576 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.041559 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 18 00:40:40.041630 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.041579 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 18 00:40:40.041716 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.041699 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 18 00:40:40.042240 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.041797 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 18 00:40:40.042240 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.041816 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 18 00:40:40.042504 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:40.042468 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 18 00:40:40.042609 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:40.042528 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-239.ec2.internal\" not found" Jul 18 00:40:40.141132 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.141083 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 18 00:40:40.142409 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.142379 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 00:40:40.142779 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.142758 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 18 00:40:40.142779 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.142793 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 18 00:40:40.142949 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.142815 2580 kubelet.go:2448] "Starting kubelet main sync loop" Jul 18 00:40:40.142949 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:40.142850 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 18 00:40:40.143594 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.143576 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-239.ec2.internal" event="NodeHasSufficientMemory" Jul 18 00:40:40.143696 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.143618 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-239.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 00:40:40.143696 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.143635 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-239.ec2.internal" event="NodeHasSufficientPID" Jul 18 00:40:40.143696 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.143695 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-239.ec2.internal" Jul 18 00:40:40.145290 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.145272 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 00:40:40.151750 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.151695 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-239.ec2.internal" Jul 18 00:40:40.151750 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:40.151720 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-239.ec2.internal\": node \"ip-10-0-134-239.ec2.internal\" not found" Jul 18 00:40:40.191340 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:40.191305 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-239.ec2.internal\" not found" Jul 18 00:40:40.243872 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.243822 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-239.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-239.ec2.internal"] Jul 18 00:40:40.243978 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.243957 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 00:40:40.244925 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.244909 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-239.ec2.internal" event="NodeHasSufficientMemory" Jul 18 00:40:40.245010 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.244939 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-239.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 00:40:40.245010 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.244948 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-239.ec2.internal" event="NodeHasSufficientPID" Jul 18 00:40:40.246214 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.246203 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 00:40:40.246348 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.246334 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-239.ec2.internal" Jul 18 00:40:40.246411 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.246369 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 00:40:40.247004 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.246987 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-239.ec2.internal" event="NodeHasSufficientMemory" Jul 18 00:40:40.247075 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.247018 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-239.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 00:40:40.247075 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.247031 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-239.ec2.internal" event="NodeHasSufficientPID" Jul 18 00:40:40.247328 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.247309 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-239.ec2.internal" event="NodeHasSufficientMemory" Jul 18 00:40:40.247385 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.247334 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-239.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 00:40:40.247385 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.247345 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-239.ec2.internal" event="NodeHasSufficientPID" Jul 18 00:40:40.248179 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.248164 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-239.ec2.internal" Jul 18 00:40:40.248258 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.248189 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 00:40:40.248823 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.248806 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-239.ec2.internal" event="NodeHasSufficientMemory" Jul 18 00:40:40.248914 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.248832 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-239.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 00:40:40.248914 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.248843 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-239.ec2.internal" event="NodeHasSufficientPID" Jul 18 00:40:40.272497 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:40.272461 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-239.ec2.internal\" not found" node="ip-10-0-134-239.ec2.internal" Jul 18 00:40:40.276882 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:40.276863 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-239.ec2.internal\" not found" node="ip-10-0-134-239.ec2.internal" Jul 18 00:40:40.281109 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.281091 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5a4e24bea70bd8be42d8d0325e1d8ad2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-239.ec2.internal\" (UID: \"5a4e24bea70bd8be42d8d0325e1d8ad2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-239.ec2.internal" Jul 18 00:40:40.281194 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.281116 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a4e24bea70bd8be42d8d0325e1d8ad2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-239.ec2.internal\" (UID: \"5a4e24bea70bd8be42d8d0325e1d8ad2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-239.ec2.internal" Jul 18 00:40:40.281194 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.281135 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/481ee2577fe0b9fe2f48bf6abc750a85-config\") pod \"kube-apiserver-proxy-ip-10-0-134-239.ec2.internal\" (UID: \"481ee2577fe0b9fe2f48bf6abc750a85\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-239.ec2.internal" Jul 18 00:40:40.292182 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:40.292155 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-239.ec2.internal\" not found" Jul 18 00:40:40.382121 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.382093 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a4e24bea70bd8be42d8d0325e1d8ad2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-239.ec2.internal\" (UID: \"5a4e24bea70bd8be42d8d0325e1d8ad2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-239.ec2.internal" Jul 18 00:40:40.382258 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.382126 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/481ee2577fe0b9fe2f48bf6abc750a85-config\") pod \"kube-apiserver-proxy-ip-10-0-134-239.ec2.internal\" (UID: \"481ee2577fe0b9fe2f48bf6abc750a85\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-239.ec2.internal" Jul 18 00:40:40.382258 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.382162 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5a4e24bea70bd8be42d8d0325e1d8ad2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-239.ec2.internal\" (UID: \"5a4e24bea70bd8be42d8d0325e1d8ad2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-239.ec2.internal" Jul 18 00:40:40.382258 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.382193 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5a4e24bea70bd8be42d8d0325e1d8ad2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-239.ec2.internal\" (UID: \"5a4e24bea70bd8be42d8d0325e1d8ad2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-239.ec2.internal" Jul 18 00:40:40.382258 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.382201 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a4e24bea70bd8be42d8d0325e1d8ad2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-239.ec2.internal\" (UID: \"5a4e24bea70bd8be42d8d0325e1d8ad2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-239.ec2.internal" Jul 18 00:40:40.382258 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.382203 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/481ee2577fe0b9fe2f48bf6abc750a85-config\") pod \"kube-apiserver-proxy-ip-10-0-134-239.ec2.internal\" (UID: \"481ee2577fe0b9fe2f48bf6abc750a85\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-239.ec2.internal" Jul 18 00:40:40.393225 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:40.393201 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-239.ec2.internal\" not found" Jul 18 00:40:40.494211 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:40.494144 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-239.ec2.internal\" not found" Jul 18 00:40:40.574382 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.574352 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-239.ec2.internal" Jul 18 00:40:40.580290 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.580272 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-239.ec2.internal" Jul 18 00:40:40.594623 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:40.594596 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-239.ec2.internal\" not found" Jul 18 00:40:40.695238 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:40.695200 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-239.ec2.internal\" not found" Jul 18 00:40:40.795784 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:40.795707 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-239.ec2.internal\" not found" Jul 18 00:40:40.896587 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:40.896547 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-239.ec2.internal\" not found" Jul 18 00:40:40.921756 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.921728 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 18 00:40:40.922240 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.921897 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 18 00:40:40.992838 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.992132 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-17 00:35:40 +0000 UTC" deadline="2028-04-16 23:48:41.930492172 +0000 UTC" Jul 18 00:40:40.992838 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:40.992170 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15335h8m0.938327373s" Jul 18 00:40:40.996999 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:40.996979 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-239.ec2.internal\" not found" Jul 18 00:40:41.040019 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.039992 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 00:40:41.098098 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:41.098065 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-239.ec2.internal\" not found" Jul 18 00:40:41.099220 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:41.099179 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4e24bea70bd8be42d8d0325e1d8ad2.slice/crio-260dbbf02e90d255acd3711f83f6003039319e8d027e95a2a8befb00ed328f7b WatchSource:0}: Error finding container 260dbbf02e90d255acd3711f83f6003039319e8d027e95a2a8befb00ed328f7b: Status 404 returned error can't find the container with id 260dbbf02e90d255acd3711f83f6003039319e8d027e95a2a8befb00ed328f7b Jul 18 00:40:41.099639 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:41.099613 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod481ee2577fe0b9fe2f48bf6abc750a85.slice/crio-3f256204410fe2308c31bd3f55cf35e3748c8cb1fad06c55e36e7836735a1492 WatchSource:0}: Error finding container 3f256204410fe2308c31bd3f55cf35e3748c8cb1fad06c55e36e7836735a1492: Status 404 returned error can't find the container with id 3f256204410fe2308c31bd3f55cf35e3748c8cb1fad06c55e36e7836735a1492 Jul 18 00:40:41.106420 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.106401 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 00:40:41.146502 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.146433 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-239.ec2.internal" event={"ID":"481ee2577fe0b9fe2f48bf6abc750a85","Type":"ContainerStarted","Data":"3f256204410fe2308c31bd3f55cf35e3748c8cb1fad06c55e36e7836735a1492"} Jul 18 00:40:41.147525 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.147502 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-239.ec2.internal" event={"ID":"5a4e24bea70bd8be42d8d0325e1d8ad2","Type":"ContainerStarted","Data":"260dbbf02e90d255acd3711f83f6003039319e8d027e95a2a8befb00ed328f7b"} Jul 18 00:40:41.179487 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.179457 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 00:40:41.192713 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.192670 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 00:40:41.280691 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.280627 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-239.ec2.internal" Jul 18 00:40:41.293126 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.293100 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 18 00:40:41.294085 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.294067 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-239.ec2.internal" Jul 18 00:40:41.300832 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.300808 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 18 00:40:41.746524 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.746488 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 00:40:41.963150 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.963116 2580 apiserver.go:52] "Watching apiserver" Jul 18 00:40:41.971901 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.971862 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 18 00:40:41.972310 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.972283 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-kj5ss","openshift-network-diagnostics/network-check-target-f5gjj","openshift-ovn-kubernetes/ovnkube-node-cwk9l","kube-system/global-pull-secret-syncer-xltqc","kube-system/konnectivity-agent-mktkv","kube-system/kube-apiserver-proxy-ip-10-0-134-239.ec2.internal","openshift-dns/node-resolver-59t57","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-239.ec2.internal","openshift-multus/multus-gh5tf","openshift-network-operator/iptables-alerter-4w8zv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq7xf","openshift-cluster-node-tuning-operator/tuned-l2wjd","openshift-image-registry/node-ca-lpl5x","openshift-multus/multus-additional-cni-plugins-qkpwq"] Jul 18 00:40:41.974466 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.973848 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq7xf" Jul 18 00:40:41.975815 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.975787 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:40:41.975921 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:41.975880 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5gjj" podUID="52500c71-f212-4425-a056-470bad08263d" Jul 18 00:40:41.976262 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.976243 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 18 00:40:41.976353 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.976336 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gnhs4\"" Jul 18 00:40:41.977183 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.976992 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 18 00:40:41.977183 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.977028 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 18 00:40:41.977461 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.977422 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:41.978771 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.978752 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:40:41.978867 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:41.978821 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj5ss" podUID="e616fcde-1cc6-4b83-b568-b0ac7076a7b3" Jul 18 00:40:41.979533 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.979513 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 18 00:40:41.980149 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.980124 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mktkv" Jul 18 00:40:41.980894 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.980872 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 18 00:40:41.981044 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.981027 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 18 00:40:41.981131 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.981115 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 18 00:40:41.981209 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.981191 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hck74\"" Jul 18 00:40:41.981270 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.981241 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 18 00:40:41.981321 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.981269 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 18 00:40:41.982327 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.982308 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qkpwq" Jul 18 00:40:41.982914 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.982896 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 18 00:40:41.983965 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.983452 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lpl5x" Jul 18 00:40:41.983965 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.983534 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mk9d7\"" Jul 18 00:40:41.983965 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.983612 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 18 00:40:41.983965 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.983762 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 18 00:40:41.985439 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.985003 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 18 00:40:41.985439 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.985083 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 18 00:40:41.985439 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.985136 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 18 00:40:41.985439 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.985224 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 18 00:40:41.985439 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.985252 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4k68p\"" Jul 18 00:40:41.985439 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.985334 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 18 00:40:41.985712 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.985592 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:41.985894 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.985875 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 18 00:40:41.986236 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.986211 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 18 00:40:41.987929 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.987910 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 18 00:40:41.988491 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.988473 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tmh7q\"" Jul 18 00:40:41.989619 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.989597 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 18 00:40:41.992794 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.990010 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 18 00:40:41.992794 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.990764 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cmbnm\"" Jul 18 00:40:41.992794 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.992060 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gh5tf" Jul 18 00:40:41.993518 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.993491 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84a3cf66-469c-4068-83c3-c6517f8d9344-ovnkube-config\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:41.993518 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.993516 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:40:41.993678 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.993541 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/835892d2-c56e-4e66-8795-13454d439aae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cq7xf\" (UID: \"835892d2-c56e-4e66-8795-13454d439aae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq7xf" Jul 18 00:40:41.993678 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.993580 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/835892d2-c56e-4e66-8795-13454d439aae-registration-dir\") pod \"aws-ebs-csi-driver-node-cq7xf\" (UID: \"835892d2-c56e-4e66-8795-13454d439aae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq7xf" Jul 18 00:40:41.993678 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:41.993592 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xltqc" podUID="3cb92a57-b715-4664-8342-fe08ec474cc1" Jul 18 00:40:41.993678 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.993615 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84a3cf66-469c-4068-83c3-c6517f8d9344-env-overrides\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:41.993678 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.993645 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e616fcde-1cc6-4b83-b568-b0ac7076a7b3-metrics-certs\") pod \"network-metrics-daemon-kj5ss\" (UID: \"e616fcde-1cc6-4b83-b568-b0ac7076a7b3\") " pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:40:41.993991 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.993723 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw7cz\" (UniqueName: \"kubernetes.io/projected/835892d2-c56e-4e66-8795-13454d439aae-kube-api-access-nw7cz\") pod \"aws-ebs-csi-driver-node-cq7xf\" (UID: \"835892d2-c56e-4e66-8795-13454d439aae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq7xf" Jul 18 00:40:41.993991 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.993756 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkgzs\" (UniqueName: \"kubernetes.io/projected/52500c71-f212-4425-a056-470bad08263d-kube-api-access-hkgzs\") pod \"network-check-target-f5gjj\" (UID: \"52500c71-f212-4425-a056-470bad08263d\") " pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:40:41.993991 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.993787 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-run-ovn\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:41.993991 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.993819 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-log-socket\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:41.993991 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.993853 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-host-cni-bin\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:41.994238 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.994178 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4w8zv" Jul 18 00:40:41.994652 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.994603 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 18 00:40:41.994652 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.994618 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84a3cf66-469c-4068-83c3-c6517f8d9344-ovnkube-script-lib\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:41.994805 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.994694 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/08f02b87-a5b3-44ef-81bb-37637a148986-agent-certs\") pod \"konnectivity-agent-mktkv\" (UID: \"08f02b87-a5b3-44ef-81bb-37637a148986\") " pod="kube-system/konnectivity-agent-mktkv" Jul 18 00:40:41.994805 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.994733 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/08f02b87-a5b3-44ef-81bb-37637a148986-konnectivity-ca\") pod \"konnectivity-agent-mktkv\" (UID: \"08f02b87-a5b3-44ef-81bb-37637a148986\") " pod="kube-system/konnectivity-agent-mktkv" Jul 18 00:40:41.994805 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.994801 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84a3cf66-469c-4068-83c3-c6517f8d9344-ovn-node-metrics-cert\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:41.996488 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.995349 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8hbc5\"" Jul 18 00:40:41.996488 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.995669 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-59t57" Jul 18 00:40:41.996488 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.995875 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r84c2\" (UniqueName: \"kubernetes.io/projected/84a3cf66-469c-4068-83c3-c6517f8d9344-kube-api-access-r84c2\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:41.996488 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.995912 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/835892d2-c56e-4e66-8795-13454d439aae-etc-selinux\") pod \"aws-ebs-csi-driver-node-cq7xf\" (UID: \"835892d2-c56e-4e66-8795-13454d439aae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq7xf" Jul 18 00:40:41.996488 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.995939 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-host-run-netns\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:41.996488 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.995965 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:41.996488 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.996013 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wglt5\" (UniqueName: \"kubernetes.io/projected/e616fcde-1cc6-4b83-b568-b0ac7076a7b3-kube-api-access-wglt5\") pod \"network-metrics-daemon-kj5ss\" (UID: \"e616fcde-1cc6-4b83-b568-b0ac7076a7b3\") " pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:40:41.996488 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.996039 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/835892d2-c56e-4e66-8795-13454d439aae-socket-dir\") pod \"aws-ebs-csi-driver-node-cq7xf\" (UID: \"835892d2-c56e-4e66-8795-13454d439aae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq7xf" Jul 18 00:40:41.996488 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.996065 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-systemd-units\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:41.996488 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.996106 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-host-slash\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:41.996488 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.996157 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-var-lib-openvswitch\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:41.996488 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.996215 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-host-cni-netd\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:41.996488 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.996251 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/835892d2-c56e-4e66-8795-13454d439aae-sys-fs\") pod \"aws-ebs-csi-driver-node-cq7xf\" (UID: \"835892d2-c56e-4e66-8795-13454d439aae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq7xf" Jul 18 00:40:41.996488 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.996286 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-run-openvswitch\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:41.996488 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.996316 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-host-run-ovn-kubernetes\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:41.996488 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.996343 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 18 00:40:41.996488 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.996349 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/835892d2-c56e-4e66-8795-13454d439aae-device-dir\") pod \"aws-ebs-csi-driver-node-cq7xf\" (UID: \"835892d2-c56e-4e66-8795-13454d439aae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq7xf" Jul 18 00:40:41.996488 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.996407 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-host-kubelet\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:41.997500 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.996428 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-run-systemd\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:41.997500 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.996442 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-etc-openvswitch\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:41.997500 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.996457 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-node-log\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:41.998339 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.998319 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-45rfj\"" Jul 18 00:40:41.998435 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.998399 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 18 00:40:41.998558 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:41.998536 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 18 00:40:42.000042 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.000022 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 18 00:40:42.000800 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.000336 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 18 00:40:42.000800 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.000573 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wwjr5\"" Jul 18 00:40:42.001250 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.001232 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 18 00:40:42.024218 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.024185 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rc4zv" Jul 18 00:40:42.029304 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.029284 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rc4zv" Jul 18 00:40:42.081330 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.081303 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 18 00:40:42.096992 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.096953 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-host-cni-bin\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.096992 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.096987 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84a3cf66-469c-4068-83c3-c6517f8d9344-ovnkube-script-lib\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.097227 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097028 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-os-release\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.097227 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097076 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-host-cni-bin\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.097227 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097103 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5aacdace-aa5e-468b-b796-89e534a2bf9d-etc-tuned\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.097227 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097139 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-multus-conf-dir\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.097227 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097184 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3cb92a57-b715-4664-8342-fe08ec474cc1-kubelet-config\") pod \"global-pull-secret-syncer-xltqc\" (UID: \"3cb92a57-b715-4664-8342-fe08ec474cc1\") " pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:40:42.097227 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097212 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/835892d2-c56e-4e66-8795-13454d439aae-etc-selinux\") pod \"aws-ebs-csi-driver-node-cq7xf\" (UID: \"835892d2-c56e-4e66-8795-13454d439aae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq7xf" Jul 18 00:40:42.097515 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097238 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wglt5\" (UniqueName: \"kubernetes.io/projected/e616fcde-1cc6-4b83-b568-b0ac7076a7b3-kube-api-access-wglt5\") pod \"network-metrics-daemon-kj5ss\" (UID: \"e616fcde-1cc6-4b83-b568-b0ac7076a7b3\") " pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:40:42.097515 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097260 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-system-cni-dir\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.097515 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097293 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6616a85b-1ee0-492b-8edc-a50d7d416d9d-cni-binary-copy\") pod \"multus-additional-cni-plugins-qkpwq\" (UID: \"6616a85b-1ee0-492b-8edc-a50d7d416d9d\") " pod="openshift-multus/multus-additional-cni-plugins-qkpwq" Jul 18 00:40:42.097515 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097343 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-var-lib-kubelet\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.097515 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097371 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3cb92a57-b715-4664-8342-fe08ec474cc1-dbus\") pod \"global-pull-secret-syncer-xltqc\" (UID: \"3cb92a57-b715-4664-8342-fe08ec474cc1\") " pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:40:42.097515 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097403 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/835892d2-c56e-4e66-8795-13454d439aae-socket-dir\") pod \"aws-ebs-csi-driver-node-cq7xf\" (UID: \"835892d2-c56e-4e66-8795-13454d439aae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq7xf" Jul 18 00:40:42.097515 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097432 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-var-lib-openvswitch\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.097515 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097427 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/835892d2-c56e-4e66-8795-13454d439aae-etc-selinux\") pod \"aws-ebs-csi-driver-node-cq7xf\" (UID: \"835892d2-c56e-4e66-8795-13454d439aae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq7xf" Jul 18 00:40:42.097515 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097468 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-host-var-lib-kubelet\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.097515 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097504 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ed6ea7dc-3703-4727-8b9d-b8671f2e89fb-serviceca\") pod \"node-ca-lpl5x\" (UID: \"ed6ea7dc-3703-4727-8b9d-b8671f2e89fb\") " pod="openshift-image-registry/node-ca-lpl5x" Jul 18 00:40:42.098135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097522 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-var-lib-openvswitch\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.098135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097533 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-etc-sysconfig\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.098135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097562 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xscm\" (UniqueName: \"kubernetes.io/projected/39f310f5-0a89-4292-90bb-5ddac09ad826-kube-api-access-5xscm\") pod \"iptables-alerter-4w8zv\" (UID: \"39f310f5-0a89-4292-90bb-5ddac09ad826\") " pod="openshift-network-operator/iptables-alerter-4w8zv" Jul 18 00:40:42.098135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097589 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/835892d2-c56e-4e66-8795-13454d439aae-sys-fs\") pod \"aws-ebs-csi-driver-node-cq7xf\" (UID: \"835892d2-c56e-4e66-8795-13454d439aae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq7xf" Jul 18 00:40:42.098135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097605 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-cnibin\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.098135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097593 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/835892d2-c56e-4e66-8795-13454d439aae-socket-dir\") pod \"aws-ebs-csi-driver-node-cq7xf\" (UID: \"835892d2-c56e-4e66-8795-13454d439aae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq7xf" Jul 18 00:40:42.098135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097627 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2159fdd6-af24-4d1b-ac04-2f8e41735fc3-hosts-file\") pod \"node-resolver-59t57\" (UID: \"2159fdd6-af24-4d1b-ac04-2f8e41735fc3\") " pod="openshift-dns/node-resolver-59t57" Jul 18 00:40:42.098135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097649 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed6ea7dc-3703-4727-8b9d-b8671f2e89fb-host\") pod \"node-ca-lpl5x\" (UID: \"ed6ea7dc-3703-4727-8b9d-b8671f2e89fb\") " pod="openshift-image-registry/node-ca-lpl5x" Jul 18 00:40:42.098135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097697 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/835892d2-c56e-4e66-8795-13454d439aae-sys-fs\") pod \"aws-ebs-csi-driver-node-cq7xf\" (UID: \"835892d2-c56e-4e66-8795-13454d439aae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq7xf" Jul 18 00:40:42.098135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097703 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/835892d2-c56e-4e66-8795-13454d439aae-device-dir\") pod \"aws-ebs-csi-driver-node-cq7xf\" (UID: \"835892d2-c56e-4e66-8795-13454d439aae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq7xf" Jul 18 00:40:42.098135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097738 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-host-kubelet\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.098135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097801 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/835892d2-c56e-4e66-8795-13454d439aae-device-dir\") pod \"aws-ebs-csi-driver-node-cq7xf\" (UID: \"835892d2-c56e-4e66-8795-13454d439aae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq7xf" Jul 18 00:40:42.098135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097832 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-run-systemd\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.098135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097850 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-host-kubelet\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.098135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097852 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/84a3cf66-469c-4068-83c3-c6517f8d9344-ovnkube-script-lib\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.098135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097902 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-run-systemd\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.098135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097928 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-etc-openvswitch\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.099135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097954 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-node-log\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.099135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097979 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84a3cf66-469c-4068-83c3-c6517f8d9344-ovnkube-config\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.099135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.097991 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-etc-openvswitch\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.099135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.098002 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-node-log\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.099135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.098006 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-hostroot\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.099135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.098045 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr7jf\" (UniqueName: \"kubernetes.io/projected/2336c951-9bb7-46e7-b596-95fca9aa7f6a-kube-api-access-lr7jf\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.099135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.098076 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-multus-cni-dir\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.099135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.098173 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2336c951-9bb7-46e7-b596-95fca9aa7f6a-cni-binary-copy\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.099135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.098196 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6616a85b-1ee0-492b-8edc-a50d7d416d9d-system-cni-dir\") pod \"multus-additional-cni-plugins-qkpwq\" (UID: \"6616a85b-1ee0-492b-8edc-a50d7d416d9d\") " pod="openshift-multus/multus-additional-cni-plugins-qkpwq" Jul 18 00:40:42.099135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.098213 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-etc-kubernetes\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.099135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.098230 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/39f310f5-0a89-4292-90bb-5ddac09ad826-iptables-alerter-script\") pod \"iptables-alerter-4w8zv\" (UID: \"39f310f5-0a89-4292-90bb-5ddac09ad826\") " pod="openshift-network-operator/iptables-alerter-4w8zv" Jul 18 00:40:42.099135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.098269 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e616fcde-1cc6-4b83-b568-b0ac7076a7b3-metrics-certs\") pod \"network-metrics-daemon-kj5ss\" (UID: \"e616fcde-1cc6-4b83-b568-b0ac7076a7b3\") " pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:40:42.099135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.098316 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-host-run-k8s-cni-cncf-io\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.099135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.098361 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6616a85b-1ee0-492b-8edc-a50d7d416d9d-os-release\") pod \"multus-additional-cni-plugins-qkpwq\" (UID: \"6616a85b-1ee0-492b-8edc-a50d7d416d9d\") " pod="openshift-multus/multus-additional-cni-plugins-qkpwq" Jul 18 00:40:42.099135 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:42.098399 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 00:40:42.099135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.098404 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw7cz\" (UniqueName: \"kubernetes.io/projected/835892d2-c56e-4e66-8795-13454d439aae-kube-api-access-nw7cz\") pod \"aws-ebs-csi-driver-node-cq7xf\" (UID: \"835892d2-c56e-4e66-8795-13454d439aae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq7xf" Jul 18 00:40:42.099135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.098422 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkgzs\" (UniqueName: \"kubernetes.io/projected/52500c71-f212-4425-a056-470bad08263d-kube-api-access-hkgzs\") pod \"network-check-target-f5gjj\" (UID: \"52500c71-f212-4425-a056-470bad08263d\") " pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:40:42.100032 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:42.098466 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e616fcde-1cc6-4b83-b568-b0ac7076a7b3-metrics-certs podName:e616fcde-1cc6-4b83-b568-b0ac7076a7b3 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:42.598432038 +0000 UTC m=+2.950282364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e616fcde-1cc6-4b83-b568-b0ac7076a7b3-metrics-certs") pod "network-metrics-daemon-kj5ss" (UID: "e616fcde-1cc6-4b83-b568-b0ac7076a7b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 00:40:42.100032 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.098470 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/84a3cf66-469c-4068-83c3-c6517f8d9344-ovnkube-config\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.100032 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.098480 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/08f02b87-a5b3-44ef-81bb-37637a148986-agent-certs\") pod \"konnectivity-agent-mktkv\" (UID: \"08f02b87-a5b3-44ef-81bb-37637a148986\") " pod="kube-system/konnectivity-agent-mktkv" Jul 18 00:40:42.100032 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.098502 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/08f02b87-a5b3-44ef-81bb-37637a148986-konnectivity-ca\") pod \"konnectivity-agent-mktkv\" (UID: \"08f02b87-a5b3-44ef-81bb-37637a148986\") " pod="kube-system/konnectivity-agent-mktkv" Jul 18 00:40:42.100032 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.098554 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ghsm\" (UniqueName: \"kubernetes.io/projected/2159fdd6-af24-4d1b-ac04-2f8e41735fc3-kube-api-access-5ghsm\") pod \"node-resolver-59t57\" (UID: \"2159fdd6-af24-4d1b-ac04-2f8e41735fc3\") " pod="openshift-dns/node-resolver-59t57" Jul 18 00:40:42.100032 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.098584 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6616a85b-1ee0-492b-8edc-a50d7d416d9d-cnibin\") pod \"multus-additional-cni-plugins-qkpwq\" (UID: \"6616a85b-1ee0-492b-8edc-a50d7d416d9d\") " pod="openshift-multus/multus-additional-cni-plugins-qkpwq" Jul 18 00:40:42.100032 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.098755 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-etc-systemd\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.100032 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.098816 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-lib-modules\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.100032 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.098842 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 18 00:40:42.100032 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.098851 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84a3cf66-469c-4068-83c3-c6517f8d9344-ovn-node-metrics-cert\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.100032 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.098922 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r84c2\" (UniqueName: \"kubernetes.io/projected/84a3cf66-469c-4068-83c3-c6517f8d9344-kube-api-access-r84c2\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.100032 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099016 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-host-var-lib-cni-bin\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.100032 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099047 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6616a85b-1ee0-492b-8edc-a50d7d416d9d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qkpwq\" (UID: \"6616a85b-1ee0-492b-8edc-a50d7d416d9d\") " pod="openshift-multus/multus-additional-cni-plugins-qkpwq" Jul 18 00:40:42.100032 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099091 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8mvb\" (UniqueName: \"kubernetes.io/projected/6616a85b-1ee0-492b-8edc-a50d7d416d9d-kube-api-access-r8mvb\") pod \"multus-additional-cni-plugins-qkpwq\" (UID: \"6616a85b-1ee0-492b-8edc-a50d7d416d9d\") " pod="openshift-multus/multus-additional-cni-plugins-qkpwq" Jul 18 00:40:42.100032 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099166 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-sys\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.100032 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099176 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/08f02b87-a5b3-44ef-81bb-37637a148986-konnectivity-ca\") pod \"konnectivity-agent-mktkv\" (UID: \"08f02b87-a5b3-44ef-81bb-37637a148986\") " pod="kube-system/konnectivity-agent-mktkv" Jul 18 00:40:42.100940 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099214 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-host-run-netns\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.100940 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099289 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.100940 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099308 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2159fdd6-af24-4d1b-ac04-2f8e41735fc3-tmp-dir\") pod \"node-resolver-59t57\" (UID: \"2159fdd6-af24-4d1b-ac04-2f8e41735fc3\") " pod="openshift-dns/node-resolver-59t57" Jul 18 00:40:42.100940 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099306 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-host-run-netns\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.100940 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099323 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-etc-modprobe-d\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.100940 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099337 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-etc-sysctl-conf\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.100940 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099354 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.100940 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099398 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-systemd-units\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.100940 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099428 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-host-slash\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.100940 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099448 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-host-cni-netd\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.100940 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099471 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-etc-kubernetes\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.100940 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099489 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-host-slash\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.100940 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099506 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-systemd-units\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.100940 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099503 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-run-openvswitch\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.100940 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099528 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-host-cni-netd\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.100940 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099546 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-run-openvswitch\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.100940 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099557 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-host-run-ovn-kubernetes\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.101715 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099582 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-multus-socket-dir-parent\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.101715 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099605 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22kf5\" (UniqueName: \"kubernetes.io/projected/ed6ea7dc-3703-4727-8b9d-b8671f2e89fb-kube-api-access-22kf5\") pod \"node-ca-lpl5x\" (UID: \"ed6ea7dc-3703-4727-8b9d-b8671f2e89fb\") " pod="openshift-image-registry/node-ca-lpl5x" Jul 18 00:40:42.101715 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099626 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-host-run-netns\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.101715 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099650 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2336c951-9bb7-46e7-b596-95fca9aa7f6a-multus-daemon-config\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.101715 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099657 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-host-run-ovn-kubernetes\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.101715 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099677 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6616a85b-1ee0-492b-8edc-a50d7d416d9d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qkpwq\" (UID: \"6616a85b-1ee0-492b-8edc-a50d7d416d9d\") " pod="openshift-multus/multus-additional-cni-plugins-qkpwq" Jul 18 00:40:42.101715 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099729 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6616a85b-1ee0-492b-8edc-a50d7d416d9d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qkpwq\" (UID: \"6616a85b-1ee0-492b-8edc-a50d7d416d9d\") " pod="openshift-multus/multus-additional-cni-plugins-qkpwq" Jul 18 00:40:42.101715 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099759 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/835892d2-c56e-4e66-8795-13454d439aae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cq7xf\" (UID: \"835892d2-c56e-4e66-8795-13454d439aae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq7xf" Jul 18 00:40:42.101715 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099786 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-host-var-lib-cni-multus\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.101715 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099965 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/835892d2-c56e-4e66-8795-13454d439aae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cq7xf\" (UID: \"835892d2-c56e-4e66-8795-13454d439aae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq7xf" Jul 18 00:40:42.101715 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.099911 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-etc-sysctl-d\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.101715 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.100088 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-host\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.101715 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.100113 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5aacdace-aa5e-468b-b796-89e534a2bf9d-tmp\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.101715 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.100134 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j998j\" (UniqueName: \"kubernetes.io/projected/5aacdace-aa5e-468b-b796-89e534a2bf9d-kube-api-access-j998j\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.101715 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.100166 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/835892d2-c56e-4e66-8795-13454d439aae-registration-dir\") pod \"aws-ebs-csi-driver-node-cq7xf\" (UID: \"835892d2-c56e-4e66-8795-13454d439aae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq7xf" Jul 18 00:40:42.101715 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.100192 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84a3cf66-469c-4068-83c3-c6517f8d9344-env-overrides\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.102404 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.100298 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-host-run-multus-certs\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.102404 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.100309 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/835892d2-c56e-4e66-8795-13454d439aae-registration-dir\") pod \"aws-ebs-csi-driver-node-cq7xf\" (UID: \"835892d2-c56e-4e66-8795-13454d439aae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq7xf" Jul 18 00:40:42.102404 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.100335 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-run\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.102404 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.100364 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39f310f5-0a89-4292-90bb-5ddac09ad826-host-slash\") pod \"iptables-alerter-4w8zv\" (UID: \"39f310f5-0a89-4292-90bb-5ddac09ad826\") " pod="openshift-network-operator/iptables-alerter-4w8zv" Jul 18 00:40:42.102404 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.100397 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cb92a57-b715-4664-8342-fe08ec474cc1-original-pull-secret\") pod \"global-pull-secret-syncer-xltqc\" (UID: \"3cb92a57-b715-4664-8342-fe08ec474cc1\") " pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:40:42.102404 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.100426 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-run-ovn\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.102404 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.100446 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-log-socket\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.102404 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.100487 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-run-ovn\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.102404 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.100517 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/84a3cf66-469c-4068-83c3-c6517f8d9344-log-socket\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.102404 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.100681 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/84a3cf66-469c-4068-83c3-c6517f8d9344-env-overrides\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.102912 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.102607 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/84a3cf66-469c-4068-83c3-c6517f8d9344-ovn-node-metrics-cert\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.102912 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.102784 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/08f02b87-a5b3-44ef-81bb-37637a148986-agent-certs\") pod \"konnectivity-agent-mktkv\" (UID: \"08f02b87-a5b3-44ef-81bb-37637a148986\") " pod="kube-system/konnectivity-agent-mktkv" Jul 18 00:40:42.109618 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.109592 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wglt5\" (UniqueName: \"kubernetes.io/projected/e616fcde-1cc6-4b83-b568-b0ac7076a7b3-kube-api-access-wglt5\") pod \"network-metrics-daemon-kj5ss\" (UID: \"e616fcde-1cc6-4b83-b568-b0ac7076a7b3\") " pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:40:42.110447 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:42.110427 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 00:40:42.110576 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:42.110455 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 00:40:42.110576 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:42.110470 2580 projected.go:196] Error preparing data for projected volume kube-api-access-hkgzs for pod openshift-network-diagnostics/network-check-target-f5gjj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 00:40:42.110706 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:42.110690 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52500c71-f212-4425-a056-470bad08263d-kube-api-access-hkgzs podName:52500c71-f212-4425-a056-470bad08263d nodeName:}" failed. No retries permitted until 2026-07-18 00:40:42.610667687 +0000 UTC m=+2.962518026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hkgzs" (UniqueName: "kubernetes.io/projected/52500c71-f212-4425-a056-470bad08263d-kube-api-access-hkgzs") pod "network-check-target-f5gjj" (UID: "52500c71-f212-4425-a056-470bad08263d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 00:40:42.113279 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.113254 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r84c2\" (UniqueName: \"kubernetes.io/projected/84a3cf66-469c-4068-83c3-c6517f8d9344-kube-api-access-r84c2\") pod \"ovnkube-node-cwk9l\" (UID: \"84a3cf66-469c-4068-83c3-c6517f8d9344\") " pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.113370 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.113337 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw7cz\" (UniqueName: \"kubernetes.io/projected/835892d2-c56e-4e66-8795-13454d439aae-kube-api-access-nw7cz\") pod \"aws-ebs-csi-driver-node-cq7xf\" (UID: \"835892d2-c56e-4e66-8795-13454d439aae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq7xf" Jul 18 00:40:42.201586 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.201550 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ghsm\" (UniqueName: \"kubernetes.io/projected/2159fdd6-af24-4d1b-ac04-2f8e41735fc3-kube-api-access-5ghsm\") pod \"node-resolver-59t57\" (UID: \"2159fdd6-af24-4d1b-ac04-2f8e41735fc3\") " pod="openshift-dns/node-resolver-59t57" Jul 18 00:40:42.201799 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.201600 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6616a85b-1ee0-492b-8edc-a50d7d416d9d-cnibin\") pod \"multus-additional-cni-plugins-qkpwq\" (UID: \"6616a85b-1ee0-492b-8edc-a50d7d416d9d\") " pod="openshift-multus/multus-additional-cni-plugins-qkpwq" Jul 18 00:40:42.201799 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.201634 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-etc-systemd\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.201799 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.201663 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-lib-modules\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.201799 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.201715 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6616a85b-1ee0-492b-8edc-a50d7d416d9d-cnibin\") pod \"multus-additional-cni-plugins-qkpwq\" (UID: \"6616a85b-1ee0-492b-8edc-a50d7d416d9d\") " pod="openshift-multus/multus-additional-cni-plugins-qkpwq" Jul 18 00:40:42.201799 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.201731 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-host-var-lib-cni-bin\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.201799 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.201756 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6616a85b-1ee0-492b-8edc-a50d7d416d9d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qkpwq\" (UID: \"6616a85b-1ee0-492b-8edc-a50d7d416d9d\") " pod="openshift-multus/multus-additional-cni-plugins-qkpwq" Jul 18 00:40:42.202096 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.201801 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8mvb\" (UniqueName: \"kubernetes.io/projected/6616a85b-1ee0-492b-8edc-a50d7d416d9d-kube-api-access-r8mvb\") pod \"multus-additional-cni-plugins-qkpwq\" (UID: \"6616a85b-1ee0-492b-8edc-a50d7d416d9d\") " pod="openshift-multus/multus-additional-cni-plugins-qkpwq" Jul 18 00:40:42.202096 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.201849 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-sys\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.202096 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.201876 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2159fdd6-af24-4d1b-ac04-2f8e41735fc3-tmp-dir\") pod \"node-resolver-59t57\" (UID: \"2159fdd6-af24-4d1b-ac04-2f8e41735fc3\") " pod="openshift-dns/node-resolver-59t57" Jul 18 00:40:42.202096 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.201913 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-etc-modprobe-d\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.202096 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.201939 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-etc-sysctl-conf\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.202096 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.201965 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-etc-kubernetes\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.202096 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.201968 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-lib-modules\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.202096 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.201998 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-multus-socket-dir-parent\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.202096 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202030 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22kf5\" (UniqueName: \"kubernetes.io/projected/ed6ea7dc-3703-4727-8b9d-b8671f2e89fb-kube-api-access-22kf5\") pod \"node-ca-lpl5x\" (UID: \"ed6ea7dc-3703-4727-8b9d-b8671f2e89fb\") " pod="openshift-image-registry/node-ca-lpl5x" Jul 18 00:40:42.202096 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202036 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-etc-systemd\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.202096 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202053 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-host-run-netns\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.202096 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202080 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2336c951-9bb7-46e7-b596-95fca9aa7f6a-multus-daemon-config\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.202638 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202108 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6616a85b-1ee0-492b-8edc-a50d7d416d9d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qkpwq\" (UID: \"6616a85b-1ee0-492b-8edc-a50d7d416d9d\") " pod="openshift-multus/multus-additional-cni-plugins-qkpwq" Jul 18 00:40:42.202638 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202135 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6616a85b-1ee0-492b-8edc-a50d7d416d9d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qkpwq\" (UID: \"6616a85b-1ee0-492b-8edc-a50d7d416d9d\") " pod="openshift-multus/multus-additional-cni-plugins-qkpwq" Jul 18 00:40:42.202638 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202167 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-host-var-lib-cni-multus\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.202638 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202193 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-etc-sysctl-d\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.202638 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202216 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-host\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.202638 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202240 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5aacdace-aa5e-468b-b796-89e534a2bf9d-tmp\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.202638 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202263 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j998j\" (UniqueName: \"kubernetes.io/projected/5aacdace-aa5e-468b-b796-89e534a2bf9d-kube-api-access-j998j\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.202638 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202289 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-host-run-multus-certs\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.202638 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202312 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-run\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.202638 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202345 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6616a85b-1ee0-492b-8edc-a50d7d416d9d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qkpwq\" (UID: \"6616a85b-1ee0-492b-8edc-a50d7d416d9d\") " pod="openshift-multus/multus-additional-cni-plugins-qkpwq" Jul 18 00:40:42.202638 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202341 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39f310f5-0a89-4292-90bb-5ddac09ad826-host-slash\") pod \"iptables-alerter-4w8zv\" (UID: \"39f310f5-0a89-4292-90bb-5ddac09ad826\") " pod="openshift-network-operator/iptables-alerter-4w8zv" Jul 18 00:40:42.202638 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202381 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2159fdd6-af24-4d1b-ac04-2f8e41735fc3-tmp-dir\") pod \"node-resolver-59t57\" (UID: \"2159fdd6-af24-4d1b-ac04-2f8e41735fc3\") " pod="openshift-dns/node-resolver-59t57" Jul 18 00:40:42.202638 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202388 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39f310f5-0a89-4292-90bb-5ddac09ad826-host-slash\") pod \"iptables-alerter-4w8zv\" (UID: \"39f310f5-0a89-4292-90bb-5ddac09ad826\") " pod="openshift-network-operator/iptables-alerter-4w8zv" Jul 18 00:40:42.202638 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202397 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cb92a57-b715-4664-8342-fe08ec474cc1-original-pull-secret\") pod \"global-pull-secret-syncer-xltqc\" (UID: \"3cb92a57-b715-4664-8342-fe08ec474cc1\") " pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:40:42.202638 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202436 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-os-release\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.202638 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202463 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5aacdace-aa5e-468b-b796-89e534a2bf9d-etc-tuned\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.202638 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202490 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-multus-conf-dir\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.203515 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202494 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-etc-modprobe-d\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.203515 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202495 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-sys\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.203515 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202516 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3cb92a57-b715-4664-8342-fe08ec474cc1-kubelet-config\") pod \"global-pull-secret-syncer-xltqc\" (UID: \"3cb92a57-b715-4664-8342-fe08ec474cc1\") " pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:40:42.203515 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202546 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-system-cni-dir\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.203515 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202572 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-run\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.203515 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202597 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-host-var-lib-cni-bin\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.203515 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202612 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-host-run-multus-certs\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.203515 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202659 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-os-release\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.203515 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202637 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-etc-sysctl-conf\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.203515 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202719 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-system-cni-dir\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.203515 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202795 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-host-var-lib-cni-multus\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.203515 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202869 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3cb92a57-b715-4664-8342-fe08ec474cc1-kubelet-config\") pod \"global-pull-secret-syncer-xltqc\" (UID: \"3cb92a57-b715-4664-8342-fe08ec474cc1\") " pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:40:42.203515 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202922 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-multus-conf-dir\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.203515 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202928 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-etc-sysctl-d\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.203515 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202872 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-etc-kubernetes\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.203515 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:42.202940 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 00:40:42.203515 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.202966 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-host-run-netns\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.203515 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:42.203008 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cb92a57-b715-4664-8342-fe08ec474cc1-original-pull-secret podName:3cb92a57-b715-4664-8342-fe08ec474cc1 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:42.702983329 +0000 UTC m=+3.054833662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3cb92a57-b715-4664-8342-fe08ec474cc1-original-pull-secret") pod "global-pull-secret-syncer-xltqc" (UID: "3cb92a57-b715-4664-8342-fe08ec474cc1") : object "kube-system"/"original-pull-secret" not registered Jul 18 00:40:42.204434 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.203019 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-host\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.204434 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.203070 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-multus-socket-dir-parent\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.204434 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.203232 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6616a85b-1ee0-492b-8edc-a50d7d416d9d-cni-binary-copy\") pod \"multus-additional-cni-plugins-qkpwq\" (UID: \"6616a85b-1ee0-492b-8edc-a50d7d416d9d\") " pod="openshift-multus/multus-additional-cni-plugins-qkpwq" Jul 18 00:40:42.204434 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.203268 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-var-lib-kubelet\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.204434 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.203291 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3cb92a57-b715-4664-8342-fe08ec474cc1-dbus\") pod \"global-pull-secret-syncer-xltqc\" (UID: \"3cb92a57-b715-4664-8342-fe08ec474cc1\") " pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:40:42.204434 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.203316 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-host-var-lib-kubelet\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.204434 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.203341 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ed6ea7dc-3703-4727-8b9d-b8671f2e89fb-serviceca\") pod \"node-ca-lpl5x\" (UID: \"ed6ea7dc-3703-4727-8b9d-b8671f2e89fb\") " pod="openshift-image-registry/node-ca-lpl5x" Jul 18 00:40:42.204434 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.203366 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-etc-sysconfig\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.204434 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.203391 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xscm\" (UniqueName: \"kubernetes.io/projected/39f310f5-0a89-4292-90bb-5ddac09ad826-kube-api-access-5xscm\") pod \"iptables-alerter-4w8zv\" (UID: \"39f310f5-0a89-4292-90bb-5ddac09ad826\") " pod="openshift-network-operator/iptables-alerter-4w8zv" Jul 18 00:40:42.204434 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.203390 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6616a85b-1ee0-492b-8edc-a50d7d416d9d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qkpwq\" (UID: \"6616a85b-1ee0-492b-8edc-a50d7d416d9d\") " pod="openshift-multus/multus-additional-cni-plugins-qkpwq" Jul 18 00:40:42.204434 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.203427 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-cnibin\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.204434 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.203459 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2336c951-9bb7-46e7-b596-95fca9aa7f6a-multus-daemon-config\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.204434 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.203467 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2159fdd6-af24-4d1b-ac04-2f8e41735fc3-hosts-file\") pod \"node-resolver-59t57\" (UID: \"2159fdd6-af24-4d1b-ac04-2f8e41735fc3\") " pod="openshift-dns/node-resolver-59t57" Jul 18 00:40:42.204434 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.203495 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed6ea7dc-3703-4727-8b9d-b8671f2e89fb-host\") pod \"node-ca-lpl5x\" (UID: \"ed6ea7dc-3703-4727-8b9d-b8671f2e89fb\") " pod="openshift-image-registry/node-ca-lpl5x" Jul 18 00:40:42.204434 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.203483 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-cnibin\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.204434 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.203526 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-host-var-lib-kubelet\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.204434 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.203573 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2159fdd6-af24-4d1b-ac04-2f8e41735fc3-hosts-file\") pod \"node-resolver-59t57\" (UID: \"2159fdd6-af24-4d1b-ac04-2f8e41735fc3\") " pod="openshift-dns/node-resolver-59t57" Jul 18 00:40:42.204434 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.203612 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed6ea7dc-3703-4727-8b9d-b8671f2e89fb-host\") pod \"node-ca-lpl5x\" (UID: \"ed6ea7dc-3703-4727-8b9d-b8671f2e89fb\") " pod="openshift-image-registry/node-ca-lpl5x" Jul 18 00:40:42.205259 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.203641 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6616a85b-1ee0-492b-8edc-a50d7d416d9d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qkpwq\" (UID: \"6616a85b-1ee0-492b-8edc-a50d7d416d9d\") " pod="openshift-multus/multus-additional-cni-plugins-qkpwq" Jul 18 00:40:42.205259 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.203667 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-etc-sysconfig\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.205259 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.203829 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3cb92a57-b715-4664-8342-fe08ec474cc1-dbus\") pod \"global-pull-secret-syncer-xltqc\" (UID: \"3cb92a57-b715-4664-8342-fe08ec474cc1\") " pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:40:42.205259 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.203529 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-hostroot\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.205259 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.203915 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr7jf\" (UniqueName: \"kubernetes.io/projected/2336c951-9bb7-46e7-b596-95fca9aa7f6a-kube-api-access-lr7jf\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.205259 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.203952 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-multus-cni-dir\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.205259 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.203959 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ed6ea7dc-3703-4727-8b9d-b8671f2e89fb-serviceca\") pod \"node-ca-lpl5x\" (UID: \"ed6ea7dc-3703-4727-8b9d-b8671f2e89fb\") " pod="openshift-image-registry/node-ca-lpl5x" Jul 18 00:40:42.205259 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.203978 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2336c951-9bb7-46e7-b596-95fca9aa7f6a-cni-binary-copy\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.205259 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.204002 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-hostroot\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.205259 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.204003 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6616a85b-1ee0-492b-8edc-a50d7d416d9d-system-cni-dir\") pod \"multus-additional-cni-plugins-qkpwq\" (UID: \"6616a85b-1ee0-492b-8edc-a50d7d416d9d\") " pod="openshift-multus/multus-additional-cni-plugins-qkpwq" Jul 18 00:40:42.205259 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.204044 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6616a85b-1ee0-492b-8edc-a50d7d416d9d-system-cni-dir\") pod \"multus-additional-cni-plugins-qkpwq\" (UID: \"6616a85b-1ee0-492b-8edc-a50d7d416d9d\") " pod="openshift-multus/multus-additional-cni-plugins-qkpwq" Jul 18 00:40:42.205259 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.204049 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6616a85b-1ee0-492b-8edc-a50d7d416d9d-cni-binary-copy\") pod \"multus-additional-cni-plugins-qkpwq\" (UID: \"6616a85b-1ee0-492b-8edc-a50d7d416d9d\") " pod="openshift-multus/multus-additional-cni-plugins-qkpwq" Jul 18 00:40:42.205259 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.203927 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-var-lib-kubelet\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.205259 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.204042 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-etc-kubernetes\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.205259 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.204092 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5aacdace-aa5e-468b-b796-89e534a2bf9d-etc-kubernetes\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.205259 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.204102 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/39f310f5-0a89-4292-90bb-5ddac09ad826-iptables-alerter-script\") pod \"iptables-alerter-4w8zv\" (UID: \"39f310f5-0a89-4292-90bb-5ddac09ad826\") " pod="openshift-network-operator/iptables-alerter-4w8zv" Jul 18 00:40:42.205259 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.204114 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-multus-cni-dir\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.205259 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.204155 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-host-run-k8s-cni-cncf-io\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.206063 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.204182 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6616a85b-1ee0-492b-8edc-a50d7d416d9d-os-release\") pod \"multus-additional-cni-plugins-qkpwq\" (UID: \"6616a85b-1ee0-492b-8edc-a50d7d416d9d\") " pod="openshift-multus/multus-additional-cni-plugins-qkpwq" Jul 18 00:40:42.206063 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.204233 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2336c951-9bb7-46e7-b596-95fca9aa7f6a-host-run-k8s-cni-cncf-io\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.206063 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.204276 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6616a85b-1ee0-492b-8edc-a50d7d416d9d-os-release\") pod \"multus-additional-cni-plugins-qkpwq\" (UID: \"6616a85b-1ee0-492b-8edc-a50d7d416d9d\") " pod="openshift-multus/multus-additional-cni-plugins-qkpwq" Jul 18 00:40:42.206063 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.204621 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/39f310f5-0a89-4292-90bb-5ddac09ad826-iptables-alerter-script\") pod \"iptables-alerter-4w8zv\" (UID: \"39f310f5-0a89-4292-90bb-5ddac09ad826\") " pod="openshift-network-operator/iptables-alerter-4w8zv" Jul 18 00:40:42.206063 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.204623 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2336c951-9bb7-46e7-b596-95fca9aa7f6a-cni-binary-copy\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.206268 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.206098 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5aacdace-aa5e-468b-b796-89e534a2bf9d-tmp\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.206268 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.206124 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5aacdace-aa5e-468b-b796-89e534a2bf9d-etc-tuned\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.211539 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.211483 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ghsm\" (UniqueName: \"kubernetes.io/projected/2159fdd6-af24-4d1b-ac04-2f8e41735fc3-kube-api-access-5ghsm\") pod \"node-resolver-59t57\" (UID: \"2159fdd6-af24-4d1b-ac04-2f8e41735fc3\") " pod="openshift-dns/node-resolver-59t57" Jul 18 00:40:42.212924 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.212870 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j998j\" (UniqueName: \"kubernetes.io/projected/5aacdace-aa5e-468b-b796-89e534a2bf9d-kube-api-access-j998j\") pod \"tuned-l2wjd\" (UID: \"5aacdace-aa5e-468b-b796-89e534a2bf9d\") " pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.213340 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.213312 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8mvb\" (UniqueName: \"kubernetes.io/projected/6616a85b-1ee0-492b-8edc-a50d7d416d9d-kube-api-access-r8mvb\") pod \"multus-additional-cni-plugins-qkpwq\" (UID: \"6616a85b-1ee0-492b-8edc-a50d7d416d9d\") " pod="openshift-multus/multus-additional-cni-plugins-qkpwq" Jul 18 00:40:42.213585 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.213563 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22kf5\" (UniqueName: \"kubernetes.io/projected/ed6ea7dc-3703-4727-8b9d-b8671f2e89fb-kube-api-access-22kf5\") pod \"node-ca-lpl5x\" (UID: \"ed6ea7dc-3703-4727-8b9d-b8671f2e89fb\") " pod="openshift-image-registry/node-ca-lpl5x" Jul 18 00:40:42.214152 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.214128 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr7jf\" (UniqueName: \"kubernetes.io/projected/2336c951-9bb7-46e7-b596-95fca9aa7f6a-kube-api-access-lr7jf\") pod \"multus-gh5tf\" (UID: \"2336c951-9bb7-46e7-b596-95fca9aa7f6a\") " pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.214246 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.214165 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xscm\" (UniqueName: \"kubernetes.io/projected/39f310f5-0a89-4292-90bb-5ddac09ad826-kube-api-access-5xscm\") pod \"iptables-alerter-4w8zv\" (UID: \"39f310f5-0a89-4292-90bb-5ddac09ad826\") " pod="openshift-network-operator/iptables-alerter-4w8zv" Jul 18 00:40:42.296236 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.296145 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq7xf" Jul 18 00:40:42.305000 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.304974 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:40:42.313119 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.313093 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-59t57" Jul 18 00:40:42.319771 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.319752 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mktkv" Jul 18 00:40:42.327863 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.327840 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qkpwq" Jul 18 00:40:42.336008 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.335990 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lpl5x" Jul 18 00:40:42.344318 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.344298 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" Jul 18 00:40:42.351937 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.351916 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gh5tf" Jul 18 00:40:42.357358 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.357338 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4w8zv" Jul 18 00:40:42.608289 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.608195 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e616fcde-1cc6-4b83-b568-b0ac7076a7b3-metrics-certs\") pod \"network-metrics-daemon-kj5ss\" (UID: \"e616fcde-1cc6-4b83-b568-b0ac7076a7b3\") " pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:40:42.608458 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:42.608323 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 00:40:42.608458 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:42.608393 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e616fcde-1cc6-4b83-b568-b0ac7076a7b3-metrics-certs podName:e616fcde-1cc6-4b83-b568-b0ac7076a7b3 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:43.608373492 +0000 UTC m=+3.960223819 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e616fcde-1cc6-4b83-b568-b0ac7076a7b3-metrics-certs") pod "network-metrics-daemon-kj5ss" (UID: "e616fcde-1cc6-4b83-b568-b0ac7076a7b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 00:40:42.709106 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.709062 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkgzs\" (UniqueName: \"kubernetes.io/projected/52500c71-f212-4425-a056-470bad08263d-kube-api-access-hkgzs\") pod \"network-check-target-f5gjj\" (UID: \"52500c71-f212-4425-a056-470bad08263d\") " pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:40:42.709294 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:42.709122 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cb92a57-b715-4664-8342-fe08ec474cc1-original-pull-secret\") pod \"global-pull-secret-syncer-xltqc\" (UID: \"3cb92a57-b715-4664-8342-fe08ec474cc1\") " pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:40:42.709294 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:42.709237 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 00:40:42.709294 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:42.709267 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 00:40:42.709294 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:42.709285 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 00:40:42.709294 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:42.709298 2580 projected.go:196] Error preparing data for projected volume kube-api-access-hkgzs for pod openshift-network-diagnostics/network-check-target-f5gjj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 00:40:42.709511 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:42.709312 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cb92a57-b715-4664-8342-fe08ec474cc1-original-pull-secret podName:3cb92a57-b715-4664-8342-fe08ec474cc1 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:43.709290428 +0000 UTC m=+4.061140768 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3cb92a57-b715-4664-8342-fe08ec474cc1-original-pull-secret") pod "global-pull-secret-syncer-xltqc" (UID: "3cb92a57-b715-4664-8342-fe08ec474cc1") : object "kube-system"/"original-pull-secret" not registered Jul 18 00:40:42.709511 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:42.709334 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52500c71-f212-4425-a056-470bad08263d-kube-api-access-hkgzs podName:52500c71-f212-4425-a056-470bad08263d nodeName:}" failed. No retries permitted until 2026-07-18 00:40:43.709324129 +0000 UTC m=+4.061174455 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hkgzs" (UniqueName: "kubernetes.io/projected/52500c71-f212-4425-a056-470bad08263d-kube-api-access-hkgzs") pod "network-check-target-f5gjj" (UID: "52500c71-f212-4425-a056-470bad08263d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 00:40:43.032087 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:43.031988 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-17 00:35:42 +0000 UTC" deadline="2028-01-24 11:14:30.356179778 +0000 UTC" Jul 18 00:40:43.032087 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:43.032030 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13330h33m47.324155308s" Jul 18 00:40:43.150723 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:43.150668 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod835892d2_c56e_4e66_8795_13454d439aae.slice/crio-b29ed8f6f1452327a00740bc3b8ca9db37ed7c8eeaffb614d9f5740cbf1a8296 WatchSource:0}: Error finding container b29ed8f6f1452327a00740bc3b8ca9db37ed7c8eeaffb614d9f5740cbf1a8296: Status 404 returned error can't find the container with id b29ed8f6f1452327a00740bc3b8ca9db37ed7c8eeaffb614d9f5740cbf1a8296 Jul 18 00:40:43.154870 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:43.154840 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a3cf66_469c_4068_83c3_c6517f8d9344.slice/crio-f20595d4b3ec3d2e636aec8536959e4209eb1ea61f23467d4d06567dc59ed6b2 WatchSource:0}: Error finding container f20595d4b3ec3d2e636aec8536959e4209eb1ea61f23467d4d06567dc59ed6b2: Status 404 returned error can't find the container with id f20595d4b3ec3d2e636aec8536959e4209eb1ea61f23467d4d06567dc59ed6b2 Jul 18 00:40:43.157054 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:43.157022 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2336c951_9bb7_46e7_b596_95fca9aa7f6a.slice/crio-a2472aa209b24664601defc74dc9f8fca7eb9dc2bb24a046552e2015f3d92046 WatchSource:0}: Error finding container a2472aa209b24664601defc74dc9f8fca7eb9dc2bb24a046552e2015f3d92046: Status 404 returned error can't find the container with id a2472aa209b24664601defc74dc9f8fca7eb9dc2bb24a046552e2015f3d92046 Jul 18 00:40:43.159158 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:43.159138 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6616a85b_1ee0_492b_8edc_a50d7d416d9d.slice/crio-92efe916cdee7f2d36b24d0f0f308746178d772d4a562ff60ed8f8eb01bf0d55 WatchSource:0}: Error finding container 92efe916cdee7f2d36b24d0f0f308746178d772d4a562ff60ed8f8eb01bf0d55: Status 404 returned error can't find the container with id 92efe916cdee7f2d36b24d0f0f308746178d772d4a562ff60ed8f8eb01bf0d55 Jul 18 00:40:43.159419 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:43.159390 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f310f5_0a89_4292_90bb_5ddac09ad826.slice/crio-a714bec2bec39f43258b09b6857e50e4f5f98b489c60ea4770d4eadff25da8e4 WatchSource:0}: Error finding container a714bec2bec39f43258b09b6857e50e4f5f98b489c60ea4770d4eadff25da8e4: Status 404 returned error can't find the container with id a714bec2bec39f43258b09b6857e50e4f5f98b489c60ea4770d4eadff25da8e4 Jul 18 00:40:43.160294 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:43.160252 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aacdace_aa5e_468b_b796_89e534a2bf9d.slice/crio-de8da2e8d30eee3168fe1b8fd996b707320f202e7f2907f66e189d778bb8d73c WatchSource:0}: Error finding container de8da2e8d30eee3168fe1b8fd996b707320f202e7f2907f66e189d778bb8d73c: Status 404 returned error can't find the container with id de8da2e8d30eee3168fe1b8fd996b707320f202e7f2907f66e189d778bb8d73c Jul 18 00:40:43.162137 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:43.162101 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2159fdd6_af24_4d1b_ac04_2f8e41735fc3.slice/crio-9bb787957c95271b2329f773c2443523bb1bbe008c6a0a03dd0e4fb0425aa7f0 WatchSource:0}: Error finding container 9bb787957c95271b2329f773c2443523bb1bbe008c6a0a03dd0e4fb0425aa7f0: Status 404 returned error can't find the container with id 9bb787957c95271b2329f773c2443523bb1bbe008c6a0a03dd0e4fb0425aa7f0 Jul 18 00:40:43.162835 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:43.162811 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f02b87_a5b3_44ef_81bb_37637a148986.slice/crio-fa9ceae714d8bace5f55c74e90ae9d04352cdfebedd493b110900fab597f5ec9 WatchSource:0}: Error finding container fa9ceae714d8bace5f55c74e90ae9d04352cdfebedd493b110900fab597f5ec9: Status 404 returned error can't find the container with id fa9ceae714d8bace5f55c74e90ae9d04352cdfebedd493b110900fab597f5ec9 Jul 18 00:40:43.188295 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:40:43.188263 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded6ea7dc_3703_4727_8b9d_b8671f2e89fb.slice/crio-9bb630b2a3b29c16b1e94f2f902b3cd5023cf0e765a7e5bfe174e0e5f3c25e4d WatchSource:0}: Error finding container 9bb630b2a3b29c16b1e94f2f902b3cd5023cf0e765a7e5bfe174e0e5f3c25e4d: Status 404 returned error can't find the container with id 9bb630b2a3b29c16b1e94f2f902b3cd5023cf0e765a7e5bfe174e0e5f3c25e4d Jul 18 00:40:43.617240 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:43.616806 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e616fcde-1cc6-4b83-b568-b0ac7076a7b3-metrics-certs\") pod \"network-metrics-daemon-kj5ss\" (UID: \"e616fcde-1cc6-4b83-b568-b0ac7076a7b3\") " pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:40:43.617240 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:43.617009 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 00:40:43.617509 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:43.617250 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e616fcde-1cc6-4b83-b568-b0ac7076a7b3-metrics-certs podName:e616fcde-1cc6-4b83-b568-b0ac7076a7b3 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:45.61722255 +0000 UTC m=+5.969072899 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e616fcde-1cc6-4b83-b568-b0ac7076a7b3-metrics-certs") pod "network-metrics-daemon-kj5ss" (UID: "e616fcde-1cc6-4b83-b568-b0ac7076a7b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 00:40:43.718621 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:43.718569 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkgzs\" (UniqueName: \"kubernetes.io/projected/52500c71-f212-4425-a056-470bad08263d-kube-api-access-hkgzs\") pod \"network-check-target-f5gjj\" (UID: \"52500c71-f212-4425-a056-470bad08263d\") " pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:40:43.718816 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:43.718656 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cb92a57-b715-4664-8342-fe08ec474cc1-original-pull-secret\") pod \"global-pull-secret-syncer-xltqc\" (UID: \"3cb92a57-b715-4664-8342-fe08ec474cc1\") " pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:40:43.718897 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:43.718821 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 00:40:43.718897 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:43.718889 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cb92a57-b715-4664-8342-fe08ec474cc1-original-pull-secret podName:3cb92a57-b715-4664-8342-fe08ec474cc1 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:45.718868579 +0000 UTC m=+6.070718910 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3cb92a57-b715-4664-8342-fe08ec474cc1-original-pull-secret") pod "global-pull-secret-syncer-xltqc" (UID: "3cb92a57-b715-4664-8342-fe08ec474cc1") : object "kube-system"/"original-pull-secret" not registered Jul 18 00:40:43.719368 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:43.719335 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 00:40:43.719368 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:43.719358 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 00:40:43.719368 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:43.719372 2580 projected.go:196] Error preparing data for projected volume kube-api-access-hkgzs for pod openshift-network-diagnostics/network-check-target-f5gjj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 00:40:43.719548 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:43.719420 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52500c71-f212-4425-a056-470bad08263d-kube-api-access-hkgzs podName:52500c71-f212-4425-a056-470bad08263d nodeName:}" failed. No retries permitted until 2026-07-18 00:40:45.719405179 +0000 UTC m=+6.071255509 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hkgzs" (UniqueName: "kubernetes.io/projected/52500c71-f212-4425-a056-470bad08263d-kube-api-access-hkgzs") pod "network-check-target-f5gjj" (UID: "52500c71-f212-4425-a056-470bad08263d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 00:40:44.033244 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:44.033123 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-17 00:35:42 +0000 UTC" deadline="2027-12-15 13:40:22.241089848 +0000 UTC" Jul 18 00:40:44.033244 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:44.033191 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12372h59m38.207904254s" Jul 18 00:40:44.145896 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:44.145861 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:40:44.146073 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:44.146007 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xltqc" podUID="3cb92a57-b715-4664-8342-fe08ec474cc1" Jul 18 00:40:44.146458 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:44.146434 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:40:44.146590 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:44.146557 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5gjj" podUID="52500c71-f212-4425-a056-470bad08263d" Jul 18 00:40:44.146661 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:44.146639 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:40:44.146776 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:44.146727 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj5ss" podUID="e616fcde-1cc6-4b83-b568-b0ac7076a7b3" Jul 18 00:40:44.175624 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:44.175582 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gh5tf" event={"ID":"2336c951-9bb7-46e7-b596-95fca9aa7f6a","Type":"ContainerStarted","Data":"a2472aa209b24664601defc74dc9f8fca7eb9dc2bb24a046552e2015f3d92046"} Jul 18 00:40:44.188964 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:44.188928 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" event={"ID":"5aacdace-aa5e-468b-b796-89e534a2bf9d","Type":"ContainerStarted","Data":"de8da2e8d30eee3168fe1b8fd996b707320f202e7f2907f66e189d778bb8d73c"} Jul 18 00:40:44.198205 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:44.198165 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" event={"ID":"84a3cf66-469c-4068-83c3-c6517f8d9344","Type":"ContainerStarted","Data":"f20595d4b3ec3d2e636aec8536959e4209eb1ea61f23467d4d06567dc59ed6b2"} Jul 18 00:40:44.209225 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:44.209127 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq7xf" event={"ID":"835892d2-c56e-4e66-8795-13454d439aae","Type":"ContainerStarted","Data":"b29ed8f6f1452327a00740bc3b8ca9db37ed7c8eeaffb614d9f5740cbf1a8296"} Jul 18 00:40:44.215773 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:44.215462 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-239.ec2.internal" event={"ID":"481ee2577fe0b9fe2f48bf6abc750a85","Type":"ContainerStarted","Data":"bfb15c979175e99df4d4687bbaf93b793cadfcceff5b6c6534bd52c388cac1dd"} Jul 18 00:40:44.231367 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:44.230959 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-239.ec2.internal" event={"ID":"5a4e24bea70bd8be42d8d0325e1d8ad2","Type":"ContainerDied","Data":"8026efe891250487ec99860732086795f0fa64bb74be4ed6152e2007957cf3ae"} Jul 18 00:40:44.231367 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:44.231340 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-239.ec2.internal" podStartSLOduration=3.231322177 podStartE2EDuration="3.231322177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:40:44.231022929 +0000 UTC m=+4.582873278" watchObservedRunningTime="2026-07-18 00:40:44.231322177 +0000 UTC m=+4.583172528" Jul 18 00:40:44.231757 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:44.230737 2580 generic.go:356] "Generic (PLEG): container finished" podID="5a4e24bea70bd8be42d8d0325e1d8ad2" containerID="8026efe891250487ec99860732086795f0fa64bb74be4ed6152e2007957cf3ae" exitCode=0 Jul 18 00:40:44.251597 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:44.251499 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lpl5x" event={"ID":"ed6ea7dc-3703-4727-8b9d-b8671f2e89fb","Type":"ContainerStarted","Data":"9bb630b2a3b29c16b1e94f2f902b3cd5023cf0e765a7e5bfe174e0e5f3c25e4d"} Jul 18 00:40:44.261914 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:44.261798 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkpwq" event={"ID":"6616a85b-1ee0-492b-8edc-a50d7d416d9d","Type":"ContainerStarted","Data":"92efe916cdee7f2d36b24d0f0f308746178d772d4a562ff60ed8f8eb01bf0d55"} Jul 18 00:40:44.298693 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:44.298630 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mktkv" event={"ID":"08f02b87-a5b3-44ef-81bb-37637a148986","Type":"ContainerStarted","Data":"fa9ceae714d8bace5f55c74e90ae9d04352cdfebedd493b110900fab597f5ec9"} Jul 18 00:40:44.306600 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:44.306562 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-59t57" event={"ID":"2159fdd6-af24-4d1b-ac04-2f8e41735fc3","Type":"ContainerStarted","Data":"9bb787957c95271b2329f773c2443523bb1bbe008c6a0a03dd0e4fb0425aa7f0"} Jul 18 00:40:44.318438 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:44.318398 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4w8zv" event={"ID":"39f310f5-0a89-4292-90bb-5ddac09ad826","Type":"ContainerStarted","Data":"a714bec2bec39f43258b09b6857e50e4f5f98b489c60ea4770d4eadff25da8e4"} Jul 18 00:40:45.347048 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:45.346820 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-239.ec2.internal" event={"ID":"5a4e24bea70bd8be42d8d0325e1d8ad2","Type":"ContainerStarted","Data":"d5f6a2fe693d606fb3d10320bee086b56b6b748392ecfefd2b42f390170993f1"} Jul 18 00:40:45.362903 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:45.362803 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-239.ec2.internal" podStartSLOduration=4.362778292 podStartE2EDuration="4.362778292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:40:45.3618357 +0000 UTC m=+5.713686049" watchObservedRunningTime="2026-07-18 00:40:45.362778292 +0000 UTC m=+5.714628643" Jul 18 00:40:45.636827 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:45.636345 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e616fcde-1cc6-4b83-b568-b0ac7076a7b3-metrics-certs\") pod \"network-metrics-daemon-kj5ss\" (UID: \"e616fcde-1cc6-4b83-b568-b0ac7076a7b3\") " pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:40:45.636827 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:45.636535 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 00:40:45.636827 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:45.636605 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e616fcde-1cc6-4b83-b568-b0ac7076a7b3-metrics-certs podName:e616fcde-1cc6-4b83-b568-b0ac7076a7b3 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:49.63658416 +0000 UTC m=+9.988434504 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e616fcde-1cc6-4b83-b568-b0ac7076a7b3-metrics-certs") pod "network-metrics-daemon-kj5ss" (UID: "e616fcde-1cc6-4b83-b568-b0ac7076a7b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 00:40:45.737707 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:45.737659 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkgzs\" (UniqueName: \"kubernetes.io/projected/52500c71-f212-4425-a056-470bad08263d-kube-api-access-hkgzs\") pod \"network-check-target-f5gjj\" (UID: \"52500c71-f212-4425-a056-470bad08263d\") " pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:40:45.737978 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:45.737756 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cb92a57-b715-4664-8342-fe08ec474cc1-original-pull-secret\") pod \"global-pull-secret-syncer-xltqc\" (UID: \"3cb92a57-b715-4664-8342-fe08ec474cc1\") " pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:40:45.737978 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:45.737924 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 00:40:45.738110 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:45.738000 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cb92a57-b715-4664-8342-fe08ec474cc1-original-pull-secret podName:3cb92a57-b715-4664-8342-fe08ec474cc1 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:49.737977116 +0000 UTC m=+10.089827456 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3cb92a57-b715-4664-8342-fe08ec474cc1-original-pull-secret") pod "global-pull-secret-syncer-xltqc" (UID: "3cb92a57-b715-4664-8342-fe08ec474cc1") : object "kube-system"/"original-pull-secret" not registered Jul 18 00:40:45.738470 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:45.738445 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 00:40:45.738579 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:45.738475 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 00:40:45.738579 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:45.738491 2580 projected.go:196] Error preparing data for projected volume kube-api-access-hkgzs for pod openshift-network-diagnostics/network-check-target-f5gjj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 00:40:45.738579 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:45.738541 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52500c71-f212-4425-a056-470bad08263d-kube-api-access-hkgzs podName:52500c71-f212-4425-a056-470bad08263d nodeName:}" failed. No retries permitted until 2026-07-18 00:40:49.738523324 +0000 UTC m=+10.090373662 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hkgzs" (UniqueName: "kubernetes.io/projected/52500c71-f212-4425-a056-470bad08263d-kube-api-access-hkgzs") pod "network-check-target-f5gjj" (UID: "52500c71-f212-4425-a056-470bad08263d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 00:40:46.152629 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:46.152600 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:40:46.152825 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:46.152765 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj5ss" podUID="e616fcde-1cc6-4b83-b568-b0ac7076a7b3" Jul 18 00:40:46.153167 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:46.153142 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:40:46.153266 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:46.153241 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xltqc" podUID="3cb92a57-b715-4664-8342-fe08ec474cc1" Jul 18 00:40:46.153331 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:46.153293 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:40:46.153381 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:46.153362 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5gjj" podUID="52500c71-f212-4425-a056-470bad08263d" Jul 18 00:40:48.143840 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:48.143800 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:40:48.143840 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:48.143856 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:40:48.144418 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:48.143800 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:40:48.144418 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:48.143951 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5gjj" podUID="52500c71-f212-4425-a056-470bad08263d" Jul 18 00:40:48.144418 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:48.144063 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xltqc" podUID="3cb92a57-b715-4664-8342-fe08ec474cc1" Jul 18 00:40:48.144418 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:48.144157 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj5ss" podUID="e616fcde-1cc6-4b83-b568-b0ac7076a7b3" Jul 18 00:40:49.675566 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:49.675526 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e616fcde-1cc6-4b83-b568-b0ac7076a7b3-metrics-certs\") pod \"network-metrics-daemon-kj5ss\" (UID: \"e616fcde-1cc6-4b83-b568-b0ac7076a7b3\") " pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:40:49.676080 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:49.675672 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 00:40:49.676080 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:49.675765 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e616fcde-1cc6-4b83-b568-b0ac7076a7b3-metrics-certs podName:e616fcde-1cc6-4b83-b568-b0ac7076a7b3 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:57.675741545 +0000 UTC m=+18.027591871 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e616fcde-1cc6-4b83-b568-b0ac7076a7b3-metrics-certs") pod "network-metrics-daemon-kj5ss" (UID: "e616fcde-1cc6-4b83-b568-b0ac7076a7b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 00:40:49.776109 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:49.775964 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkgzs\" (UniqueName: \"kubernetes.io/projected/52500c71-f212-4425-a056-470bad08263d-kube-api-access-hkgzs\") pod \"network-check-target-f5gjj\" (UID: \"52500c71-f212-4425-a056-470bad08263d\") " pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:40:49.776109 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:49.776037 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cb92a57-b715-4664-8342-fe08ec474cc1-original-pull-secret\") pod \"global-pull-secret-syncer-xltqc\" (UID: \"3cb92a57-b715-4664-8342-fe08ec474cc1\") " pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:40:49.776372 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:49.776143 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 00:40:49.776372 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:49.776163 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 00:40:49.776372 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:49.776192 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 00:40:49.776372 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:49.776207 2580 projected.go:196] Error preparing data for projected volume kube-api-access-hkgzs for pod openshift-network-diagnostics/network-check-target-f5gjj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 00:40:49.776372 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:49.776222 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cb92a57-b715-4664-8342-fe08ec474cc1-original-pull-secret podName:3cb92a57-b715-4664-8342-fe08ec474cc1 nodeName:}" failed. No retries permitted until 2026-07-18 00:40:57.776204515 +0000 UTC m=+18.128054864 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3cb92a57-b715-4664-8342-fe08ec474cc1-original-pull-secret") pod "global-pull-secret-syncer-xltqc" (UID: "3cb92a57-b715-4664-8342-fe08ec474cc1") : object "kube-system"/"original-pull-secret" not registered Jul 18 00:40:49.776372 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:49.776254 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52500c71-f212-4425-a056-470bad08263d-kube-api-access-hkgzs podName:52500c71-f212-4425-a056-470bad08263d nodeName:}" failed. No retries permitted until 2026-07-18 00:40:57.776241491 +0000 UTC m=+18.128091815 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hkgzs" (UniqueName: "kubernetes.io/projected/52500c71-f212-4425-a056-470bad08263d-kube-api-access-hkgzs") pod "network-check-target-f5gjj" (UID: "52500c71-f212-4425-a056-470bad08263d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 00:40:50.144843 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:50.144745 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:40:50.144843 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:50.144763 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:40:50.144843 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:50.144776 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:40:50.145133 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:50.144908 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xltqc" podUID="3cb92a57-b715-4664-8342-fe08ec474cc1" Jul 18 00:40:50.145133 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:50.144974 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5gjj" podUID="52500c71-f212-4425-a056-470bad08263d" Jul 18 00:40:50.145133 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:50.145065 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj5ss" podUID="e616fcde-1cc6-4b83-b568-b0ac7076a7b3" Jul 18 00:40:52.148141 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:52.148105 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:40:52.148734 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:52.148245 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj5ss" podUID="e616fcde-1cc6-4b83-b568-b0ac7076a7b3" Jul 18 00:40:52.148734 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:52.148614 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:40:52.148734 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:52.148726 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xltqc" podUID="3cb92a57-b715-4664-8342-fe08ec474cc1" Jul 18 00:40:52.148907 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:52.148784 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:40:52.148907 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:52.148846 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5gjj" podUID="52500c71-f212-4425-a056-470bad08263d" Jul 18 00:40:54.143911 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:54.143868 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:40:54.143911 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:54.143906 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:40:54.144385 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:54.143878 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:40:54.144385 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:54.144009 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xltqc" podUID="3cb92a57-b715-4664-8342-fe08ec474cc1" Jul 18 00:40:54.144385 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:54.144067 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5gjj" podUID="52500c71-f212-4425-a056-470bad08263d" Jul 18 00:40:54.144385 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:54.144141 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj5ss" podUID="e616fcde-1cc6-4b83-b568-b0ac7076a7b3" Jul 18 00:40:56.143134 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:56.143095 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:40:56.143134 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:56.143135 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:40:56.143695 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:56.143234 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xltqc" podUID="3cb92a57-b715-4664-8342-fe08ec474cc1" Jul 18 00:40:56.143695 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:56.143305 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5gjj" podUID="52500c71-f212-4425-a056-470bad08263d" Jul 18 00:40:56.143695 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:56.143357 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:40:56.143695 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:56.143407 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj5ss" podUID="e616fcde-1cc6-4b83-b568-b0ac7076a7b3" Jul 18 00:40:57.735861 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:57.735826 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e616fcde-1cc6-4b83-b568-b0ac7076a7b3-metrics-certs\") pod \"network-metrics-daemon-kj5ss\" (UID: \"e616fcde-1cc6-4b83-b568-b0ac7076a7b3\") " pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:40:57.736461 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:57.736021 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 00:40:57.736461 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:57.736105 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e616fcde-1cc6-4b83-b568-b0ac7076a7b3-metrics-certs podName:e616fcde-1cc6-4b83-b568-b0ac7076a7b3 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:13.736083119 +0000 UTC m=+34.087933459 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e616fcde-1cc6-4b83-b568-b0ac7076a7b3-metrics-certs") pod "network-metrics-daemon-kj5ss" (UID: "e616fcde-1cc6-4b83-b568-b0ac7076a7b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 00:40:57.836335 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:57.836300 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkgzs\" (UniqueName: \"kubernetes.io/projected/52500c71-f212-4425-a056-470bad08263d-kube-api-access-hkgzs\") pod \"network-check-target-f5gjj\" (UID: \"52500c71-f212-4425-a056-470bad08263d\") " pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:40:57.836519 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:57.836365 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cb92a57-b715-4664-8342-fe08ec474cc1-original-pull-secret\") pod \"global-pull-secret-syncer-xltqc\" (UID: \"3cb92a57-b715-4664-8342-fe08ec474cc1\") " pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:40:57.836519 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:57.836482 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 00:40:57.836519 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:57.836494 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 00:40:57.836707 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:57.836524 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 00:40:57.836707 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:57.836537 2580 projected.go:196] Error preparing data for projected volume kube-api-access-hkgzs for pod openshift-network-diagnostics/network-check-target-f5gjj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 00:40:57.836707 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:57.836551 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cb92a57-b715-4664-8342-fe08ec474cc1-original-pull-secret podName:3cb92a57-b715-4664-8342-fe08ec474cc1 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:13.836532769 +0000 UTC m=+34.188383100 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3cb92a57-b715-4664-8342-fe08ec474cc1-original-pull-secret") pod "global-pull-secret-syncer-xltqc" (UID: "3cb92a57-b715-4664-8342-fe08ec474cc1") : object "kube-system"/"original-pull-secret" not registered Jul 18 00:40:57.836707 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:57.836589 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52500c71-f212-4425-a056-470bad08263d-kube-api-access-hkgzs podName:52500c71-f212-4425-a056-470bad08263d nodeName:}" failed. No retries permitted until 2026-07-18 00:41:13.836572759 +0000 UTC m=+34.188423100 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hkgzs" (UniqueName: "kubernetes.io/projected/52500c71-f212-4425-a056-470bad08263d-kube-api-access-hkgzs") pod "network-check-target-f5gjj" (UID: "52500c71-f212-4425-a056-470bad08263d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 00:40:58.143601 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:58.143522 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:40:58.143601 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:58.143546 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:40:58.143827 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:58.143648 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj5ss" podUID="e616fcde-1cc6-4b83-b568-b0ac7076a7b3" Jul 18 00:40:58.143827 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:40:58.143684 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:40:58.143827 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:58.143763 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5gjj" podUID="52500c71-f212-4425-a056-470bad08263d" Jul 18 00:40:58.143950 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:40:58.143837 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xltqc" podUID="3cb92a57-b715-4664-8342-fe08ec474cc1" Jul 18 00:41:00.144639 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:00.144177 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:41:00.144639 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:00.144278 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:41:00.144639 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:00.144283 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xltqc" podUID="3cb92a57-b715-4664-8342-fe08ec474cc1" Jul 18 00:41:00.144639 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:00.144400 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5gjj" podUID="52500c71-f212-4425-a056-470bad08263d" Jul 18 00:41:00.144639 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:00.144434 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:41:00.144639 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:00.144534 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj5ss" podUID="e616fcde-1cc6-4b83-b568-b0ac7076a7b3" Jul 18 00:41:02.143314 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:02.143275 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:41:02.143877 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:02.143279 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:41:02.143877 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:02.143410 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5gjj" podUID="52500c71-f212-4425-a056-470bad08263d" Jul 18 00:41:02.143877 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:02.143513 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xltqc" podUID="3cb92a57-b715-4664-8342-fe08ec474cc1" Jul 18 00:41:02.143877 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:02.143294 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:41:02.143877 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:02.143626 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj5ss" podUID="e616fcde-1cc6-4b83-b568-b0ac7076a7b3" Jul 18 00:41:03.387752 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:03.387506 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gh5tf" event={"ID":"2336c951-9bb7-46e7-b596-95fca9aa7f6a","Type":"ContainerStarted","Data":"f45b724b08c4b905d1729beee50570e339d55a3c9f3da94274cd166e289ed482"} Jul 18 00:41:03.389219 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:03.389189 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" event={"ID":"5aacdace-aa5e-468b-b796-89e534a2bf9d","Type":"ContainerStarted","Data":"9ad4fcf15ca28b637151cfcafb8ce9486a520852c394467830ed71af4255a725"} Jul 18 00:41:03.390852 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:03.390830 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" event={"ID":"84a3cf66-469c-4068-83c3-c6517f8d9344","Type":"ContainerStarted","Data":"e67cd0e472512f3d669000e3f3b104bc4bbc93ed55e088f46918d5b522367524"} Jul 18 00:41:03.392565 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:03.392541 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq7xf" event={"ID":"835892d2-c56e-4e66-8795-13454d439aae","Type":"ContainerStarted","Data":"4f1b7332007356ba34790d3d546c20369235035bfe048b06af1328905f12a5af"} Jul 18 00:41:03.393900 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:03.393877 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lpl5x" event={"ID":"ed6ea7dc-3703-4727-8b9d-b8671f2e89fb","Type":"ContainerStarted","Data":"fa5451fb0ae7d5237fe8356bcf6d1a95a3ac6b092c6ece5f01afdcb2fed76d40"} Jul 18 00:41:03.395368 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:03.395346 2580 generic.go:356] "Generic (PLEG): container finished" podID="6616a85b-1ee0-492b-8edc-a50d7d416d9d" containerID="f207b851e695963379327f12ca8e401431e35773f0fb2f2289792207ef384299" exitCode=0 Jul 18 00:41:03.395480 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:03.395405 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkpwq" event={"ID":"6616a85b-1ee0-492b-8edc-a50d7d416d9d","Type":"ContainerDied","Data":"f207b851e695963379327f12ca8e401431e35773f0fb2f2289792207ef384299"} Jul 18 00:41:03.397024 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:03.397003 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mktkv" event={"ID":"08f02b87-a5b3-44ef-81bb-37637a148986","Type":"ContainerStarted","Data":"9d915fd4fe16d3a554a69d094fb47dbd8d9cfda9d92a1c36ac1eb751b817a79d"} Jul 18 00:41:03.398334 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:03.398316 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-59t57" event={"ID":"2159fdd6-af24-4d1b-ac04-2f8e41735fc3","Type":"ContainerStarted","Data":"58161d4942f62cd1b75290b0ee225187fded0407c98ccdfa2d7c4be7d0fb53ca"} Jul 18 00:41:03.406482 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:03.406436 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gh5tf" podStartSLOduration=3.460618212 podStartE2EDuration="23.406419507s" totalImagesPullingTime="19.945801295s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:03.405877768 +0000 UTC m=+23.757728114" watchObservedRunningTime="2026-07-18 00:41:03.406419507 +0000 UTC m=+23.758269854" Jul 18 00:41:03.492266 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:03.492030 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lpl5x" podStartSLOduration=3.625948445 podStartE2EDuration="23.492009028s" totalImagesPullingTime="19.866060583s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:03.438534914 +0000 UTC m=+23.790385261" watchObservedRunningTime="2026-07-18 00:41:03.492009028 +0000 UTC m=+23.843859557" Jul 18 00:41:03.508925 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:03.508870 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-59t57" podStartSLOduration=2.61402587 podStartE2EDuration="22.508854048s" totalImagesPullingTime="19.894828178s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:03.508526721 +0000 UTC m=+23.860377068" watchObservedRunningTime="2026-07-18 00:41:03.508854048 +0000 UTC m=+23.860704392" Jul 18 00:41:03.509179 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:03.509153 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-l2wjd" podStartSLOduration=3.635810009 podStartE2EDuration="23.509145315s" totalImagesPullingTime="19.873335306s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:03.4932379 +0000 UTC m=+23.845088383" watchObservedRunningTime="2026-07-18 00:41:03.509145315 +0000 UTC m=+23.860995663" Jul 18 00:41:04.145624 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:04.145591 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:41:04.145814 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:04.145739 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xltqc" podUID="3cb92a57-b715-4664-8342-fe08ec474cc1" Jul 18 00:41:04.146119 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:04.146102 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:41:04.146213 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:04.146194 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5gjj" podUID="52500c71-f212-4425-a056-470bad08263d" Jul 18 00:41:04.146291 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:04.146269 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:41:04.146365 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:04.146351 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj5ss" podUID="e616fcde-1cc6-4b83-b568-b0ac7076a7b3" Jul 18 00:41:04.404299 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:04.404203 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" event={"ID":"84a3cf66-469c-4068-83c3-c6517f8d9344","Type":"ContainerStarted","Data":"e11248f249ff852c0039c42eafe668f5720ba5f538e33edda398231276c81036"} Jul 18 00:41:04.404299 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:04.404286 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" event={"ID":"84a3cf66-469c-4068-83c3-c6517f8d9344","Type":"ContainerStarted","Data":"4fc63579091235a4ab0178bdfa245ee1162e4572ff6ef20ff60c11823903b845"} Jul 18 00:41:04.405015 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:04.404347 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" event={"ID":"84a3cf66-469c-4068-83c3-c6517f8d9344","Type":"ContainerStarted","Data":"8cde06c26717ec367d113dde6bb39cd878d1313a27dd4aeb107b28284f34d5ed"} Jul 18 00:41:04.405015 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:04.404395 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" event={"ID":"84a3cf66-469c-4068-83c3-c6517f8d9344","Type":"ContainerStarted","Data":"01c6cb19c405fce4fea819d09ecc0baa3a22190cfe5046c18612381bd5f2d8a0"} Jul 18 00:41:04.405015 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:04.404441 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" event={"ID":"84a3cf66-469c-4068-83c3-c6517f8d9344","Type":"ContainerStarted","Data":"0ef5371c1b2acfb785ff549c52192d6300e2ac452c051fd947c5a9690770bbf1"} Jul 18 00:41:04.685124 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:04.685094 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 18 00:41:05.052868 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:05.052730 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-18T00:41:04.685119119Z","UUID":"193df026-1362-473e-b3b0-5c0e13ecc934","Handler":null,"Name":"","Endpoint":""} Jul 18 00:41:05.054624 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:05.054600 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 18 00:41:05.054752 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:05.054645 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 18 00:41:05.409573 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:05.408841 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4w8zv" event={"ID":"39f310f5-0a89-4292-90bb-5ddac09ad826","Type":"ContainerStarted","Data":"0c7e051f94ac303eb71c3d01549b1891ed5d1aa5dd3cba7f8d8bcc66c00f708d"} Jul 18 00:41:05.411547 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:05.411512 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq7xf" event={"ID":"835892d2-c56e-4e66-8795-13454d439aae","Type":"ContainerStarted","Data":"f73617a611d8e6038866927436c138b342d9cb6f0322b58ea3cee30a08226c3d"} Jul 18 00:41:05.432788 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:05.432732 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mktkv" podStartSLOduration=5.562141289 podStartE2EDuration="25.43271361s" totalImagesPullingTime="19.870572321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:03.524803587 +0000 UTC m=+23.876653932" watchObservedRunningTime="2026-07-18 00:41:05.43271361 +0000 UTC m=+25.784563955" Jul 18 00:41:05.433030 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:05.433005 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4w8zv" podStartSLOduration=5.532055159 podStartE2EDuration="25.432996968s" totalImagesPullingTime="19.900941809s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:05.423945553 +0000 UTC m=+25.775795900" watchObservedRunningTime="2026-07-18 00:41:05.432996968 +0000 UTC m=+25.784847312" Jul 18 00:41:06.143520 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:06.143479 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:41:06.143699 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:06.143624 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xltqc" podUID="3cb92a57-b715-4664-8342-fe08ec474cc1" Jul 18 00:41:06.143699 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:06.143479 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:41:06.143826 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:06.143484 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:41:06.143826 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:06.143802 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5gjj" podUID="52500c71-f212-4425-a056-470bad08263d" Jul 18 00:41:06.143952 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:06.143919 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj5ss" podUID="e616fcde-1cc6-4b83-b568-b0ac7076a7b3" Jul 18 00:41:06.416572 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:06.416488 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq7xf" event={"ID":"835892d2-c56e-4e66-8795-13454d439aae","Type":"ContainerStarted","Data":"5f923238f8cbc102bf526f9adce89f5ef06fd8cd0497345068667a3aa835e014"} Jul 18 00:41:06.434456 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:06.434400 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cq7xf" podStartSLOduration=4.318108647 podStartE2EDuration="26.434380887s" totalImagesPullingTime="22.11627224s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:06.433993364 +0000 UTC m=+26.785843711" watchObservedRunningTime="2026-07-18 00:41:06.434380887 +0000 UTC m=+26.786231232" Jul 18 00:41:06.994749 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:06.994701 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mktkv" Jul 18 00:41:06.995504 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:06.995482 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mktkv" Jul 18 00:41:07.418997 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:07.418922 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mktkv" Jul 18 00:41:07.419381 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:07.419367 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mktkv" Jul 18 00:41:08.143710 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:08.143510 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:41:08.143710 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:08.143510 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:41:08.143850 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:08.143510 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:41:08.143904 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:08.143843 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xltqc" podUID="3cb92a57-b715-4664-8342-fe08ec474cc1" Jul 18 00:41:08.143904 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:08.143851 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj5ss" podUID="e616fcde-1cc6-4b83-b568-b0ac7076a7b3" Jul 18 00:41:08.144006 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:08.143911 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5gjj" podUID="52500c71-f212-4425-a056-470bad08263d" Jul 18 00:41:08.422562 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:08.422483 2580 generic.go:356] "Generic (PLEG): container finished" podID="6616a85b-1ee0-492b-8edc-a50d7d416d9d" containerID="df4128beaa3d8a878fd72ef03140f50b0a865f3c549e2581525c656ce1bb39f8" exitCode=0 Jul 18 00:41:08.423105 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:08.422570 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkpwq" event={"ID":"6616a85b-1ee0-492b-8edc-a50d7d416d9d","Type":"ContainerDied","Data":"df4128beaa3d8a878fd72ef03140f50b0a865f3c549e2581525c656ce1bb39f8"} Jul 18 00:41:08.426076 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:08.426051 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" event={"ID":"84a3cf66-469c-4068-83c3-c6517f8d9344","Type":"ContainerStarted","Data":"acbfaec0a4cf3c6d4df5970e6bcc6d45bd8a7eb6a656b10b6519270f0da287cc"} Jul 18 00:41:09.429635 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:09.429602 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkpwq" event={"ID":"6616a85b-1ee0-492b-8edc-a50d7d416d9d","Type":"ContainerStarted","Data":"3f3e33c476dd5245c2ef39a86673e5fdd24b0f36bab9d49e170e3b8830299736"} Jul 18 00:41:10.144134 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:10.143952 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:41:10.144357 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:10.144026 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:41:10.144357 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:10.144050 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:41:10.144357 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:10.144279 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj5ss" podUID="e616fcde-1cc6-4b83-b568-b0ac7076a7b3" Jul 18 00:41:10.144357 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:10.144320 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xltqc" podUID="3cb92a57-b715-4664-8342-fe08ec474cc1" Jul 18 00:41:10.144357 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:10.144189 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5gjj" podUID="52500c71-f212-4425-a056-470bad08263d" Jul 18 00:41:10.433519 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:10.433432 2580 generic.go:356] "Generic (PLEG): container finished" podID="6616a85b-1ee0-492b-8edc-a50d7d416d9d" containerID="3f3e33c476dd5245c2ef39a86673e5fdd24b0f36bab9d49e170e3b8830299736" exitCode=0 Jul 18 00:41:10.434222 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:10.433518 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkpwq" event={"ID":"6616a85b-1ee0-492b-8edc-a50d7d416d9d","Type":"ContainerDied","Data":"3f3e33c476dd5245c2ef39a86673e5fdd24b0f36bab9d49e170e3b8830299736"} Jul 18 00:41:10.439206 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:10.438556 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" event={"ID":"84a3cf66-469c-4068-83c3-c6517f8d9344","Type":"ContainerStarted","Data":"6104e16548290c46ddb82bf387c43597d9a771e0a546304b23d50c0c93d4c559"} Jul 18 00:41:10.484733 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:10.484682 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" podStartSLOduration=10.480568832 podStartE2EDuration="30.484648574s" totalImagesPullingTime="20.004079742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:10.483255342 +0000 UTC m=+30.835105710" watchObservedRunningTime="2026-07-18 00:41:10.484648574 +0000 UTC m=+30.836498918" Jul 18 00:41:11.447361 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:11.447161 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:41:11.447361 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:11.447248 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:41:11.447361 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:11.447297 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:41:11.466897 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:11.466773 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:41:11.468507 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:11.468485 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:41:11.720847 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:11.720812 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kj5ss"] Jul 18 00:41:11.721026 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:11.720961 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:41:11.721098 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:11.721075 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj5ss" podUID="e616fcde-1cc6-4b83-b568-b0ac7076a7b3" Jul 18 00:41:11.729659 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:11.729475 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f5gjj"] Jul 18 00:41:11.729659 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:11.729640 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:41:11.729886 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:11.729768 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5gjj" podUID="52500c71-f212-4425-a056-470bad08263d" Jul 18 00:41:11.730216 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:11.730174 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xltqc"] Jul 18 00:41:11.730348 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:11.730280 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:41:11.730411 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:11.730387 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xltqc" podUID="3cb92a57-b715-4664-8342-fe08ec474cc1" Jul 18 00:41:12.451528 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:12.451499 2580 generic.go:356] "Generic (PLEG): container finished" podID="6616a85b-1ee0-492b-8edc-a50d7d416d9d" containerID="3165a03c796d49f8ab19ffa530eefce9963f656a8b1da65a7f163381c8bd0451" exitCode=0 Jul 18 00:41:12.451960 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:12.451581 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkpwq" event={"ID":"6616a85b-1ee0-492b-8edc-a50d7d416d9d","Type":"ContainerDied","Data":"3165a03c796d49f8ab19ffa530eefce9963f656a8b1da65a7f163381c8bd0451"} Jul 18 00:41:13.143329 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:13.143128 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:41:13.143495 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:13.143136 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:41:13.143495 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:13.143423 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xltqc" podUID="3cb92a57-b715-4664-8342-fe08ec474cc1" Jul 18 00:41:13.143608 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:13.143518 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj5ss" podUID="e616fcde-1cc6-4b83-b568-b0ac7076a7b3" Jul 18 00:41:13.143608 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:13.143169 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:41:13.143739 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:13.143619 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5gjj" podUID="52500c71-f212-4425-a056-470bad08263d" Jul 18 00:41:13.752687 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:13.752643 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e616fcde-1cc6-4b83-b568-b0ac7076a7b3-metrics-certs\") pod \"network-metrics-daemon-kj5ss\" (UID: \"e616fcde-1cc6-4b83-b568-b0ac7076a7b3\") " pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:41:13.753118 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:13.752802 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 00:41:13.753118 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:13.752881 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e616fcde-1cc6-4b83-b568-b0ac7076a7b3-metrics-certs podName:e616fcde-1cc6-4b83-b568-b0ac7076a7b3 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:45.752864954 +0000 UTC m=+66.104715276 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e616fcde-1cc6-4b83-b568-b0ac7076a7b3-metrics-certs") pod "network-metrics-daemon-kj5ss" (UID: "e616fcde-1cc6-4b83-b568-b0ac7076a7b3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 00:41:13.853489 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:13.853445 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cb92a57-b715-4664-8342-fe08ec474cc1-original-pull-secret\") pod \"global-pull-secret-syncer-xltqc\" (UID: \"3cb92a57-b715-4664-8342-fe08ec474cc1\") " pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:41:13.853633 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:13.853530 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkgzs\" (UniqueName: \"kubernetes.io/projected/52500c71-f212-4425-a056-470bad08263d-kube-api-access-hkgzs\") pod \"network-check-target-f5gjj\" (UID: \"52500c71-f212-4425-a056-470bad08263d\") " pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:41:13.853710 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:13.853625 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 00:41:13.853710 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:13.853696 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 00:41:13.853710 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:13.853710 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 00:41:13.853858 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:13.853717 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cb92a57-b715-4664-8342-fe08ec474cc1-original-pull-secret podName:3cb92a57-b715-4664-8342-fe08ec474cc1 nodeName:}" failed. No retries permitted until 2026-07-18 00:41:45.853697049 +0000 UTC m=+66.205547375 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3cb92a57-b715-4664-8342-fe08ec474cc1-original-pull-secret") pod "global-pull-secret-syncer-xltqc" (UID: "3cb92a57-b715-4664-8342-fe08ec474cc1") : object "kube-system"/"original-pull-secret" not registered Jul 18 00:41:13.853858 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:13.853720 2580 projected.go:196] Error preparing data for projected volume kube-api-access-hkgzs for pod openshift-network-diagnostics/network-check-target-f5gjj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 00:41:13.853858 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:13.853765 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52500c71-f212-4425-a056-470bad08263d-kube-api-access-hkgzs podName:52500c71-f212-4425-a056-470bad08263d nodeName:}" failed. No retries permitted until 2026-07-18 00:41:45.853752116 +0000 UTC m=+66.205602439 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hkgzs" (UniqueName: "kubernetes.io/projected/52500c71-f212-4425-a056-470bad08263d-kube-api-access-hkgzs") pod "network-check-target-f5gjj" (UID: "52500c71-f212-4425-a056-470bad08263d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 00:41:15.144124 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:15.144089 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:41:15.144637 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:15.144223 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xltqc" podUID="3cb92a57-b715-4664-8342-fe08ec474cc1" Jul 18 00:41:15.144637 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:15.144231 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:41:15.144637 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:15.144262 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:41:15.144637 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:15.144318 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f5gjj" podUID="52500c71-f212-4425-a056-470bad08263d" Jul 18 00:41:15.144637 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:15.144374 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kj5ss" podUID="e616fcde-1cc6-4b83-b568-b0ac7076a7b3" Jul 18 00:41:16.011174 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.011143 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-239.ec2.internal" event="NodeReady" Jul 18 00:41:16.011387 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.011296 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 18 00:41:16.058628 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.058590 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dvbgz"] Jul 18 00:41:16.085686 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.085637 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qpq66"] Jul 18 00:41:16.085860 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.085766 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dvbgz" Jul 18 00:41:16.088548 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.088520 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 18 00:41:16.088548 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.088536 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 18 00:41:16.088906 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.088887 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9hbbz\"" Jul 18 00:41:16.098771 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.098752 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dvbgz"] Jul 18 00:41:16.098882 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.098784 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qpq66"] Jul 18 00:41:16.098882 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.098799 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-swzld"] Jul 18 00:41:16.098978 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.098889 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qpq66" Jul 18 00:41:16.101788 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.101765 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 18 00:41:16.101904 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.101848 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 18 00:41:16.101966 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.101764 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-txq7s\"" Jul 18 00:41:16.102020 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.101997 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 18 00:41:16.115761 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.115737 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-swzld"] Jul 18 00:41:16.115895 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.115880 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-swzld" Jul 18 00:41:16.118812 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.118793 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 18 00:41:16.118921 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.118859 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 18 00:41:16.118982 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.118962 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-w2tbv\"" Jul 18 00:41:16.119037 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.118983 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 18 00:41:16.119094 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.119075 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 18 00:41:16.172580 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.172541 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7471583-1515-4d25-afeb-539e6484f43d-tmp-dir\") pod \"dns-default-dvbgz\" (UID: \"e7471583-1515-4d25-afeb-539e6484f43d\") " pod="openshift-dns/dns-default-dvbgz" Jul 18 00:41:16.173039 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.172613 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e7471583-1515-4d25-afeb-539e6484f43d-metrics-tls\") pod \"dns-default-dvbgz\" (UID: \"e7471583-1515-4d25-afeb-539e6484f43d\") " pod="openshift-dns/dns-default-dvbgz" Jul 18 00:41:16.173039 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.172634 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd525\" (UniqueName: \"kubernetes.io/projected/e7471583-1515-4d25-afeb-539e6484f43d-kube-api-access-kd525\") pod \"dns-default-dvbgz\" (UID: \"e7471583-1515-4d25-afeb-539e6484f43d\") " pod="openshift-dns/dns-default-dvbgz" Jul 18 00:41:16.173039 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.172655 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a02dce2-9348-462f-be64-820d42bdd849-cert\") pod \"ingress-canary-qpq66\" (UID: \"1a02dce2-9348-462f-be64-820d42bdd849\") " pod="openshift-ingress-canary/ingress-canary-qpq66" Jul 18 00:41:16.173039 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.172739 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6qsd\" (UniqueName: \"kubernetes.io/projected/1a02dce2-9348-462f-be64-820d42bdd849-kube-api-access-x6qsd\") pod \"ingress-canary-qpq66\" (UID: \"1a02dce2-9348-462f-be64-820d42bdd849\") " pod="openshift-ingress-canary/ingress-canary-qpq66" Jul 18 00:41:16.173039 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.172801 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e7471583-1515-4d25-afeb-539e6484f43d-config-volume\") pod \"dns-default-dvbgz\" (UID: \"e7471583-1515-4d25-afeb-539e6484f43d\") " pod="openshift-dns/dns-default-dvbgz" Jul 18 00:41:16.273790 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.273706 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a02dce2-9348-462f-be64-820d42bdd849-cert\") pod \"ingress-canary-qpq66\" (UID: \"1a02dce2-9348-462f-be64-820d42bdd849\") " pod="openshift-ingress-canary/ingress-canary-qpq66" Jul 18 00:41:16.273790 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.273745 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1fc2c61e-6d9a-4c11-b942-770a3d15153c-crio-socket\") pod \"insights-runtime-extractor-swzld\" (UID: \"1fc2c61e-6d9a-4c11-b942-770a3d15153c\") " pod="openshift-insights/insights-runtime-extractor-swzld" Jul 18 00:41:16.273790 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.273768 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgjx6\" (UniqueName: \"kubernetes.io/projected/1fc2c61e-6d9a-4c11-b942-770a3d15153c-kube-api-access-rgjx6\") pod \"insights-runtime-extractor-swzld\" (UID: \"1fc2c61e-6d9a-4c11-b942-770a3d15153c\") " pod="openshift-insights/insights-runtime-extractor-swzld" Jul 18 00:41:16.274046 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.273800 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6qsd\" (UniqueName: \"kubernetes.io/projected/1a02dce2-9348-462f-be64-820d42bdd849-kube-api-access-x6qsd\") pod \"ingress-canary-qpq66\" (UID: \"1a02dce2-9348-462f-be64-820d42bdd849\") " pod="openshift-ingress-canary/ingress-canary-qpq66" Jul 18 00:41:16.274046 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.273890 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e7471583-1515-4d25-afeb-539e6484f43d-config-volume\") pod \"dns-default-dvbgz\" (UID: \"e7471583-1515-4d25-afeb-539e6484f43d\") " pod="openshift-dns/dns-default-dvbgz" Jul 18 00:41:16.274046 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.273944 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fc2c61e-6d9a-4c11-b942-770a3d15153c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-swzld\" (UID: \"1fc2c61e-6d9a-4c11-b942-770a3d15153c\") " pod="openshift-insights/insights-runtime-extractor-swzld" Jul 18 00:41:16.274046 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.273989 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1fc2c61e-6d9a-4c11-b942-770a3d15153c-data-volume\") pod \"insights-runtime-extractor-swzld\" (UID: \"1fc2c61e-6d9a-4c11-b942-770a3d15153c\") " pod="openshift-insights/insights-runtime-extractor-swzld" Jul 18 00:41:16.274046 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.274011 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fc2c61e-6d9a-4c11-b942-770a3d15153c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-swzld\" (UID: \"1fc2c61e-6d9a-4c11-b942-770a3d15153c\") " pod="openshift-insights/insights-runtime-extractor-swzld" Jul 18 00:41:16.274214 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.274053 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7471583-1515-4d25-afeb-539e6484f43d-tmp-dir\") pod \"dns-default-dvbgz\" (UID: \"e7471583-1515-4d25-afeb-539e6484f43d\") " pod="openshift-dns/dns-default-dvbgz" Jul 18 00:41:16.274214 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.274150 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e7471583-1515-4d25-afeb-539e6484f43d-metrics-tls\") pod \"dns-default-dvbgz\" (UID: \"e7471583-1515-4d25-afeb-539e6484f43d\") " pod="openshift-dns/dns-default-dvbgz" Jul 18 00:41:16.274307 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:16.274221 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 00:41:16.274307 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:16.274289 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7471583-1515-4d25-afeb-539e6484f43d-metrics-tls podName:e7471583-1515-4d25-afeb-539e6484f43d nodeName:}" failed. No retries permitted until 2026-07-18 00:41:16.774268935 +0000 UTC m=+37.126119275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e7471583-1515-4d25-afeb-539e6484f43d-metrics-tls") pod "dns-default-dvbgz" (UID: "e7471583-1515-4d25-afeb-539e6484f43d") : secret "dns-default-metrics-tls" not found Jul 18 00:41:16.274405 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.274386 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7471583-1515-4d25-afeb-539e6484f43d-tmp-dir\") pod \"dns-default-dvbgz\" (UID: \"e7471583-1515-4d25-afeb-539e6484f43d\") " pod="openshift-dns/dns-default-dvbgz" Jul 18 00:41:16.274464 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.274448 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd525\" (UniqueName: \"kubernetes.io/projected/e7471583-1515-4d25-afeb-539e6484f43d-kube-api-access-kd525\") pod \"dns-default-dvbgz\" (UID: \"e7471583-1515-4d25-afeb-539e6484f43d\") " pod="openshift-dns/dns-default-dvbgz" Jul 18 00:41:16.274500 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.274465 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e7471583-1515-4d25-afeb-539e6484f43d-config-volume\") pod \"dns-default-dvbgz\" (UID: \"e7471583-1515-4d25-afeb-539e6484f43d\") " pod="openshift-dns/dns-default-dvbgz" Jul 18 00:41:16.278573 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.278548 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a02dce2-9348-462f-be64-820d42bdd849-cert\") pod \"ingress-canary-qpq66\" (UID: \"1a02dce2-9348-462f-be64-820d42bdd849\") " pod="openshift-ingress-canary/ingress-canary-qpq66" Jul 18 00:41:16.286153 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.286107 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd525\" (UniqueName: \"kubernetes.io/projected/e7471583-1515-4d25-afeb-539e6484f43d-kube-api-access-kd525\") pod \"dns-default-dvbgz\" (UID: \"e7471583-1515-4d25-afeb-539e6484f43d\") " pod="openshift-dns/dns-default-dvbgz" Jul 18 00:41:16.286299 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.286208 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6qsd\" (UniqueName: \"kubernetes.io/projected/1a02dce2-9348-462f-be64-820d42bdd849-kube-api-access-x6qsd\") pod \"ingress-canary-qpq66\" (UID: \"1a02dce2-9348-462f-be64-820d42bdd849\") " pod="openshift-ingress-canary/ingress-canary-qpq66" Jul 18 00:41:16.375330 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.375291 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1fc2c61e-6d9a-4c11-b942-770a3d15153c-crio-socket\") pod \"insights-runtime-extractor-swzld\" (UID: \"1fc2c61e-6d9a-4c11-b942-770a3d15153c\") " pod="openshift-insights/insights-runtime-extractor-swzld" Jul 18 00:41:16.375504 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.375339 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgjx6\" (UniqueName: \"kubernetes.io/projected/1fc2c61e-6d9a-4c11-b942-770a3d15153c-kube-api-access-rgjx6\") pod \"insights-runtime-extractor-swzld\" (UID: \"1fc2c61e-6d9a-4c11-b942-770a3d15153c\") " pod="openshift-insights/insights-runtime-extractor-swzld" Jul 18 00:41:16.375504 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.375387 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fc2c61e-6d9a-4c11-b942-770a3d15153c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-swzld\" (UID: \"1fc2c61e-6d9a-4c11-b942-770a3d15153c\") " pod="openshift-insights/insights-runtime-extractor-swzld" Jul 18 00:41:16.375504 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.375420 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1fc2c61e-6d9a-4c11-b942-770a3d15153c-data-volume\") pod \"insights-runtime-extractor-swzld\" (UID: \"1fc2c61e-6d9a-4c11-b942-770a3d15153c\") " pod="openshift-insights/insights-runtime-extractor-swzld" Jul 18 00:41:16.375504 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.375440 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fc2c61e-6d9a-4c11-b942-770a3d15153c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-swzld\" (UID: \"1fc2c61e-6d9a-4c11-b942-770a3d15153c\") " pod="openshift-insights/insights-runtime-extractor-swzld" Jul 18 00:41:16.375732 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:16.375627 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 00:41:16.375732 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:16.375710 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fc2c61e-6d9a-4c11-b942-770a3d15153c-insights-runtime-extractor-tls podName:1fc2c61e-6d9a-4c11-b942-770a3d15153c nodeName:}" failed. No retries permitted until 2026-07-18 00:41:16.87568741 +0000 UTC m=+37.227537755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1fc2c61e-6d9a-4c11-b942-770a3d15153c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-swzld" (UID: "1fc2c61e-6d9a-4c11-b942-770a3d15153c") : secret "insights-runtime-extractor-tls" not found Jul 18 00:41:16.375896 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:16.375872 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 18 00:41:16.376011 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:16.375945 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1fc2c61e-6d9a-4c11-b942-770a3d15153c-kube-rbac-proxy-cm podName:1fc2c61e-6d9a-4c11-b942-770a3d15153c nodeName:}" failed. No retries permitted until 2026-07-18 00:41:16.875927603 +0000 UTC m=+37.227777930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1fc2c61e-6d9a-4c11-b942-770a3d15153c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-swzld" (UID: "1fc2c61e-6d9a-4c11-b942-770a3d15153c") : configmap "kube-rbac-proxy" not found Jul 18 00:41:16.376075 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.376026 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1fc2c61e-6d9a-4c11-b942-770a3d15153c-crio-socket\") pod \"insights-runtime-extractor-swzld\" (UID: \"1fc2c61e-6d9a-4c11-b942-770a3d15153c\") " pod="openshift-insights/insights-runtime-extractor-swzld" Jul 18 00:41:16.376276 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.376256 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1fc2c61e-6d9a-4c11-b942-770a3d15153c-data-volume\") pod \"insights-runtime-extractor-swzld\" (UID: \"1fc2c61e-6d9a-4c11-b942-770a3d15153c\") " pod="openshift-insights/insights-runtime-extractor-swzld" Jul 18 00:41:16.389979 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.389947 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgjx6\" (UniqueName: \"kubernetes.io/projected/1fc2c61e-6d9a-4c11-b942-770a3d15153c-kube-api-access-rgjx6\") pod \"insights-runtime-extractor-swzld\" (UID: \"1fc2c61e-6d9a-4c11-b942-770a3d15153c\") " pod="openshift-insights/insights-runtime-extractor-swzld" Jul 18 00:41:16.408997 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.408965 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qpq66" Jul 18 00:41:16.661725 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.661696 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qpq66"] Jul 18 00:41:16.737682 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:41:16.737645 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a02dce2_9348_462f_be64_820d42bdd849.slice/crio-822b9a12b2074ef05da623bcc0b13cbd712b90fe49773fd9697e80c2138edd2a WatchSource:0}: Error finding container 822b9a12b2074ef05da623bcc0b13cbd712b90fe49773fd9697e80c2138edd2a: Status 404 returned error can't find the container with id 822b9a12b2074ef05da623bcc0b13cbd712b90fe49773fd9697e80c2138edd2a Jul 18 00:41:16.779274 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.779243 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e7471583-1515-4d25-afeb-539e6484f43d-metrics-tls\") pod \"dns-default-dvbgz\" (UID: \"e7471583-1515-4d25-afeb-539e6484f43d\") " pod="openshift-dns/dns-default-dvbgz" Jul 18 00:41:16.779383 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:16.779359 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 00:41:16.779438 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:16.779426 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7471583-1515-4d25-afeb-539e6484f43d-metrics-tls podName:e7471583-1515-4d25-afeb-539e6484f43d nodeName:}" failed. No retries permitted until 2026-07-18 00:41:17.779405069 +0000 UTC m=+38.131255397 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e7471583-1515-4d25-afeb-539e6484f43d-metrics-tls") pod "dns-default-dvbgz" (UID: "e7471583-1515-4d25-afeb-539e6484f43d") : secret "dns-default-metrics-tls" not found Jul 18 00:41:16.779790 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.779770 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-59t57_2159fdd6-af24-4d1b-ac04-2f8e41735fc3/dns-node-resolver/0.log" Jul 18 00:41:16.880700 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.880464 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fc2c61e-6d9a-4c11-b942-770a3d15153c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-swzld\" (UID: \"1fc2c61e-6d9a-4c11-b942-770a3d15153c\") " pod="openshift-insights/insights-runtime-extractor-swzld" Jul 18 00:41:16.880804 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:16.880730 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fc2c61e-6d9a-4c11-b942-770a3d15153c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-swzld\" (UID: \"1fc2c61e-6d9a-4c11-b942-770a3d15153c\") " pod="openshift-insights/insights-runtime-extractor-swzld" Jul 18 00:41:16.880804 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:16.880585 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 18 00:41:16.880865 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:16.880834 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 00:41:16.880865 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:16.880845 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1fc2c61e-6d9a-4c11-b942-770a3d15153c-kube-rbac-proxy-cm podName:1fc2c61e-6d9a-4c11-b942-770a3d15153c nodeName:}" failed. No retries permitted until 2026-07-18 00:41:17.880824182 +0000 UTC m=+38.232674510 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1fc2c61e-6d9a-4c11-b942-770a3d15153c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-swzld" (UID: "1fc2c61e-6d9a-4c11-b942-770a3d15153c") : configmap "kube-rbac-proxy" not found Jul 18 00:41:16.880938 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:16.880867 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fc2c61e-6d9a-4c11-b942-770a3d15153c-insights-runtime-extractor-tls podName:1fc2c61e-6d9a-4c11-b942-770a3d15153c nodeName:}" failed. No retries permitted until 2026-07-18 00:41:17.880857222 +0000 UTC m=+38.232707548 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1fc2c61e-6d9a-4c11-b942-770a3d15153c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-swzld" (UID: "1fc2c61e-6d9a-4c11-b942-770a3d15153c") : secret "insights-runtime-extractor-tls" not found Jul 18 00:41:17.034568 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.034539 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-2dlzj"] Jul 18 00:41:17.045249 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.045061 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-2dlzj" Jul 18 00:41:17.045801 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.045780 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-2dlzj"] Jul 18 00:41:17.047636 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.047615 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 18 00:41:17.047747 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.047662 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 18 00:41:17.047978 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.047965 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 18 00:41:17.048084 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.048064 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 18 00:41:17.048144 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.048113 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-qljkx\"" Jul 18 00:41:17.143153 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.143044 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:41:17.143153 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.143044 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:41:17.143375 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.143044 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:41:17.145791 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.145764 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 18 00:41:17.145901 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.145797 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 18 00:41:17.145901 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.145829 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-475p8\"" Jul 18 00:41:17.145901 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.145836 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 18 00:41:17.145901 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.145865 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d2hq6\"" Jul 18 00:41:17.145901 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.145862 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 18 00:41:17.182531 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.182499 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7d818d3b-d5ae-422e-b858-d7b322f3f7f7-signing-cabundle\") pod \"service-ca-58598695f7-2dlzj\" (UID: \"7d818d3b-d5ae-422e-b858-d7b322f3f7f7\") " pod="openshift-service-ca/service-ca-58598695f7-2dlzj" Jul 18 00:41:17.183392 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.182644 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7d818d3b-d5ae-422e-b858-d7b322f3f7f7-signing-key\") pod \"service-ca-58598695f7-2dlzj\" (UID: \"7d818d3b-d5ae-422e-b858-d7b322f3f7f7\") " pod="openshift-service-ca/service-ca-58598695f7-2dlzj" Jul 18 00:41:17.183392 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.182713 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc2kx\" (UniqueName: \"kubernetes.io/projected/7d818d3b-d5ae-422e-b858-d7b322f3f7f7-kube-api-access-mc2kx\") pod \"service-ca-58598695f7-2dlzj\" (UID: \"7d818d3b-d5ae-422e-b858-d7b322f3f7f7\") " pod="openshift-service-ca/service-ca-58598695f7-2dlzj" Jul 18 00:41:17.283949 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.283912 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7d818d3b-d5ae-422e-b858-d7b322f3f7f7-signing-key\") pod \"service-ca-58598695f7-2dlzj\" (UID: \"7d818d3b-d5ae-422e-b858-d7b322f3f7f7\") " pod="openshift-service-ca/service-ca-58598695f7-2dlzj" Jul 18 00:41:17.284126 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.283969 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc2kx\" (UniqueName: \"kubernetes.io/projected/7d818d3b-d5ae-422e-b858-d7b322f3f7f7-kube-api-access-mc2kx\") pod \"service-ca-58598695f7-2dlzj\" (UID: \"7d818d3b-d5ae-422e-b858-d7b322f3f7f7\") " pod="openshift-service-ca/service-ca-58598695f7-2dlzj" Jul 18 00:41:17.284126 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.284034 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7d818d3b-d5ae-422e-b858-d7b322f3f7f7-signing-cabundle\") pod \"service-ca-58598695f7-2dlzj\" (UID: \"7d818d3b-d5ae-422e-b858-d7b322f3f7f7\") " pod="openshift-service-ca/service-ca-58598695f7-2dlzj" Jul 18 00:41:17.284956 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.284921 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7d818d3b-d5ae-422e-b858-d7b322f3f7f7-signing-cabundle\") pod \"service-ca-58598695f7-2dlzj\" (UID: \"7d818d3b-d5ae-422e-b858-d7b322f3f7f7\") " pod="openshift-service-ca/service-ca-58598695f7-2dlzj" Jul 18 00:41:17.287543 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.287518 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7d818d3b-d5ae-422e-b858-d7b322f3f7f7-signing-key\") pod \"service-ca-58598695f7-2dlzj\" (UID: \"7d818d3b-d5ae-422e-b858-d7b322f3f7f7\") " pod="openshift-service-ca/service-ca-58598695f7-2dlzj" Jul 18 00:41:17.294490 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.294469 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc2kx\" (UniqueName: \"kubernetes.io/projected/7d818d3b-d5ae-422e-b858-d7b322f3f7f7-kube-api-access-mc2kx\") pod \"service-ca-58598695f7-2dlzj\" (UID: \"7d818d3b-d5ae-422e-b858-d7b322f3f7f7\") " pod="openshift-service-ca/service-ca-58598695f7-2dlzj" Jul 18 00:41:17.354839 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.354807 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-2dlzj" Jul 18 00:41:17.465685 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.465631 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qpq66" event={"ID":"1a02dce2-9348-462f-be64-820d42bdd849","Type":"ContainerStarted","Data":"822b9a12b2074ef05da623bcc0b13cbd712b90fe49773fd9697e80c2138edd2a"} Jul 18 00:41:17.469148 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.469123 2580 generic.go:356] "Generic (PLEG): container finished" podID="6616a85b-1ee0-492b-8edc-a50d7d416d9d" containerID="32e8c1ce092c34c0c7fa749fd5339cfb783868b5aa6aa9fcf5d4adbb5b5deea7" exitCode=0 Jul 18 00:41:17.469248 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.469157 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkpwq" event={"ID":"6616a85b-1ee0-492b-8edc-a50d7d416d9d","Type":"ContainerDied","Data":"32e8c1ce092c34c0c7fa749fd5339cfb783868b5aa6aa9fcf5d4adbb5b5deea7"} Jul 18 00:41:17.498949 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.498177 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-2dlzj"] Jul 18 00:41:17.505063 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:41:17.505033 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d818d3b_d5ae_422e_b858_d7b322f3f7f7.slice/crio-a62b1375ea168b27a84d053c07d4108bb91a83001bd30e70bdd304359336bc31 WatchSource:0}: Error finding container a62b1375ea168b27a84d053c07d4108bb91a83001bd30e70bdd304359336bc31: Status 404 returned error can't find the container with id a62b1375ea168b27a84d053c07d4108bb91a83001bd30e70bdd304359336bc31 Jul 18 00:41:17.788328 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:17.788288 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 00:41:17.790801 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.788117 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e7471583-1515-4d25-afeb-539e6484f43d-metrics-tls\") pod \"dns-default-dvbgz\" (UID: \"e7471583-1515-4d25-afeb-539e6484f43d\") " pod="openshift-dns/dns-default-dvbgz" Jul 18 00:41:17.790943 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:17.790805 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7471583-1515-4d25-afeb-539e6484f43d-metrics-tls podName:e7471583-1515-4d25-afeb-539e6484f43d nodeName:}" failed. No retries permitted until 2026-07-18 00:41:19.790777952 +0000 UTC m=+40.142628275 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e7471583-1515-4d25-afeb-539e6484f43d-metrics-tls") pod "dns-default-dvbgz" (UID: "e7471583-1515-4d25-afeb-539e6484f43d") : secret "dns-default-metrics-tls" not found Jul 18 00:41:17.891498 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.891457 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fc2c61e-6d9a-4c11-b942-770a3d15153c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-swzld\" (UID: \"1fc2c61e-6d9a-4c11-b942-770a3d15153c\") " pod="openshift-insights/insights-runtime-extractor-swzld" Jul 18 00:41:17.891498 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.891509 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fc2c61e-6d9a-4c11-b942-770a3d15153c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-swzld\" (UID: \"1fc2c61e-6d9a-4c11-b942-770a3d15153c\") " pod="openshift-insights/insights-runtime-extractor-swzld" Jul 18 00:41:17.891763 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:17.891631 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 00:41:17.891763 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:17.891630 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 18 00:41:17.891763 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:17.891722 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fc2c61e-6d9a-4c11-b942-770a3d15153c-insights-runtime-extractor-tls podName:1fc2c61e-6d9a-4c11-b942-770a3d15153c nodeName:}" failed. No retries permitted until 2026-07-18 00:41:19.891701442 +0000 UTC m=+40.243551784 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1fc2c61e-6d9a-4c11-b942-770a3d15153c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-swzld" (UID: "1fc2c61e-6d9a-4c11-b942-770a3d15153c") : secret "insights-runtime-extractor-tls" not found Jul 18 00:41:17.891924 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:17.891789 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1fc2c61e-6d9a-4c11-b942-770a3d15153c-kube-rbac-proxy-cm podName:1fc2c61e-6d9a-4c11-b942-770a3d15153c nodeName:}" failed. No retries permitted until 2026-07-18 00:41:19.891769593 +0000 UTC m=+40.243619917 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1fc2c61e-6d9a-4c11-b942-770a3d15153c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-swzld" (UID: "1fc2c61e-6d9a-4c11-b942-770a3d15153c") : configmap "kube-rbac-proxy" not found Jul 18 00:41:17.963620 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:17.963594 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lpl5x_ed6ea7dc-3703-4727-8b9d-b8671f2e89fb/node-ca/0.log" Jul 18 00:41:18.475846 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:18.475808 2580 generic.go:356] "Generic (PLEG): container finished" podID="6616a85b-1ee0-492b-8edc-a50d7d416d9d" containerID="949589757d99d4a2bf7a52fd436b4c5ce24cb10ab3c5d4908f55ab60ec0148c2" exitCode=0 Jul 18 00:41:18.476329 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:18.475887 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkpwq" event={"ID":"6616a85b-1ee0-492b-8edc-a50d7d416d9d","Type":"ContainerDied","Data":"949589757d99d4a2bf7a52fd436b4c5ce24cb10ab3c5d4908f55ab60ec0148c2"} Jul 18 00:41:18.477377 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:18.477347 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-2dlzj" event={"ID":"7d818d3b-d5ae-422e-b858-d7b322f3f7f7","Type":"ContainerStarted","Data":"a62b1375ea168b27a84d053c07d4108bb91a83001bd30e70bdd304359336bc31"} Jul 18 00:41:19.808521 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:19.808332 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e7471583-1515-4d25-afeb-539e6484f43d-metrics-tls\") pod \"dns-default-dvbgz\" (UID: \"e7471583-1515-4d25-afeb-539e6484f43d\") " pod="openshift-dns/dns-default-dvbgz" Jul 18 00:41:19.808521 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:19.808517 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 00:41:19.809066 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:19.808600 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7471583-1515-4d25-afeb-539e6484f43d-metrics-tls podName:e7471583-1515-4d25-afeb-539e6484f43d nodeName:}" failed. No retries permitted until 2026-07-18 00:41:23.808579436 +0000 UTC m=+44.160429763 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e7471583-1515-4d25-afeb-539e6484f43d-metrics-tls") pod "dns-default-dvbgz" (UID: "e7471583-1515-4d25-afeb-539e6484f43d") : secret "dns-default-metrics-tls" not found Jul 18 00:41:19.909325 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:19.909281 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fc2c61e-6d9a-4c11-b942-770a3d15153c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-swzld\" (UID: \"1fc2c61e-6d9a-4c11-b942-770a3d15153c\") " pod="openshift-insights/insights-runtime-extractor-swzld" Jul 18 00:41:19.909325 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:19.909326 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fc2c61e-6d9a-4c11-b942-770a3d15153c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-swzld\" (UID: \"1fc2c61e-6d9a-4c11-b942-770a3d15153c\") " pod="openshift-insights/insights-runtime-extractor-swzld" Jul 18 00:41:19.909583 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:19.909427 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 18 00:41:19.909583 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:19.909451 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 00:41:19.909583 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:19.909511 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1fc2c61e-6d9a-4c11-b942-770a3d15153c-kube-rbac-proxy-cm podName:1fc2c61e-6d9a-4c11-b942-770a3d15153c nodeName:}" failed. No retries permitted until 2026-07-18 00:41:23.909489821 +0000 UTC m=+44.261340148 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1fc2c61e-6d9a-4c11-b942-770a3d15153c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-swzld" (UID: "1fc2c61e-6d9a-4c11-b942-770a3d15153c") : configmap "kube-rbac-proxy" not found Jul 18 00:41:19.909583 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:19.909535 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fc2c61e-6d9a-4c11-b942-770a3d15153c-insights-runtime-extractor-tls podName:1fc2c61e-6d9a-4c11-b942-770a3d15153c nodeName:}" failed. No retries permitted until 2026-07-18 00:41:23.909523543 +0000 UTC m=+44.261373866 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1fc2c61e-6d9a-4c11-b942-770a3d15153c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-swzld" (UID: "1fc2c61e-6d9a-4c11-b942-770a3d15153c") : secret "insights-runtime-extractor-tls" not found Jul 18 00:41:20.484082 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:20.484041 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qpq66" event={"ID":"1a02dce2-9348-462f-be64-820d42bdd849","Type":"ContainerStarted","Data":"d54d0a498a1e3ecc163c92247a753095cdfaf08374f3cf39de27ec2f857f6626"} Jul 18 00:41:20.485527 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:20.485500 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-2dlzj" event={"ID":"7d818d3b-d5ae-422e-b858-d7b322f3f7f7","Type":"ContainerStarted","Data":"0f3e24680b5d8ab29882758e8043c70b8a46d179629d8767873ebbf46a6d8726"} Jul 18 00:41:20.492458 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:20.492429 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qkpwq" event={"ID":"6616a85b-1ee0-492b-8edc-a50d7d416d9d","Type":"ContainerStarted","Data":"3a0405a2da5788223e8f535c36e9abf969f9006825998fb4cb0f76ea94135869"} Jul 18 00:41:20.501403 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:20.501353 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qpq66" podStartSLOduration=1.724807939 podStartE2EDuration="4.501336991s" totalImagesPullingTime="2.776529052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:41:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:20.501058702 +0000 UTC m=+40.852909049" watchObservedRunningTime="2026-07-18 00:41:20.501336991 +0000 UTC m=+40.853187316" Jul 18 00:41:20.525889 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:20.525837 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-2dlzj" podStartSLOduration=1.5119356499999999 podStartE2EDuration="3.525821152s" totalImagesPullingTime="2.013885502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:20.5252728 +0000 UTC m=+40.877123144" watchObservedRunningTime="2026-07-18 00:41:20.525821152 +0000 UTC m=+40.877671497" Jul 18 00:41:20.550712 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:20.550638 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qkpwq" podStartSLOduration=10.121678819 podStartE2EDuration="40.550622776s" totalImagesPullingTime="30.428943957s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:20.549082387 +0000 UTC m=+40.900932732" watchObservedRunningTime="2026-07-18 00:41:20.550622776 +0000 UTC m=+40.902473120" Jul 18 00:41:23.842171 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:23.842129 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e7471583-1515-4d25-afeb-539e6484f43d-metrics-tls\") pod \"dns-default-dvbgz\" (UID: \"e7471583-1515-4d25-afeb-539e6484f43d\") " pod="openshift-dns/dns-default-dvbgz" Jul 18 00:41:23.842648 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:23.842292 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 00:41:23.842648 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:23.842370 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7471583-1515-4d25-afeb-539e6484f43d-metrics-tls podName:e7471583-1515-4d25-afeb-539e6484f43d nodeName:}" failed. No retries permitted until 2026-07-18 00:41:31.842350585 +0000 UTC m=+52.194200909 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e7471583-1515-4d25-afeb-539e6484f43d-metrics-tls") pod "dns-default-dvbgz" (UID: "e7471583-1515-4d25-afeb-539e6484f43d") : secret "dns-default-metrics-tls" not found Jul 18 00:41:23.942732 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:23.942699 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fc2c61e-6d9a-4c11-b942-770a3d15153c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-swzld\" (UID: \"1fc2c61e-6d9a-4c11-b942-770a3d15153c\") " pod="openshift-insights/insights-runtime-extractor-swzld" Jul 18 00:41:23.942901 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:23.942741 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fc2c61e-6d9a-4c11-b942-770a3d15153c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-swzld\" (UID: \"1fc2c61e-6d9a-4c11-b942-770a3d15153c\") " pod="openshift-insights/insights-runtime-extractor-swzld" Jul 18 00:41:23.942901 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:23.942826 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 18 00:41:23.942901 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:23.942878 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 00:41:23.942901 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:23.942893 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1fc2c61e-6d9a-4c11-b942-770a3d15153c-kube-rbac-proxy-cm podName:1fc2c61e-6d9a-4c11-b942-770a3d15153c nodeName:}" failed. No retries permitted until 2026-07-18 00:41:31.942875496 +0000 UTC m=+52.294725819 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1fc2c61e-6d9a-4c11-b942-770a3d15153c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-swzld" (UID: "1fc2c61e-6d9a-4c11-b942-770a3d15153c") : configmap "kube-rbac-proxy" not found Jul 18 00:41:23.943031 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:41:23.942912 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fc2c61e-6d9a-4c11-b942-770a3d15153c-insights-runtime-extractor-tls podName:1fc2c61e-6d9a-4c11-b942-770a3d15153c nodeName:}" failed. No retries permitted until 2026-07-18 00:41:31.942902979 +0000 UTC m=+52.294753303 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1fc2c61e-6d9a-4c11-b942-770a3d15153c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-swzld" (UID: "1fc2c61e-6d9a-4c11-b942-770a3d15153c") : secret "insights-runtime-extractor-tls" not found Jul 18 00:41:31.905919 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:31.905878 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e7471583-1515-4d25-afeb-539e6484f43d-metrics-tls\") pod \"dns-default-dvbgz\" (UID: \"e7471583-1515-4d25-afeb-539e6484f43d\") " pod="openshift-dns/dns-default-dvbgz" Jul 18 00:41:31.909594 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:31.909563 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e7471583-1515-4d25-afeb-539e6484f43d-metrics-tls\") pod \"dns-default-dvbgz\" (UID: \"e7471583-1515-4d25-afeb-539e6484f43d\") " pod="openshift-dns/dns-default-dvbgz" Jul 18 00:41:31.996136 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:31.996103 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dvbgz" Jul 18 00:41:32.006491 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:32.006458 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fc2c61e-6d9a-4c11-b942-770a3d15153c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-swzld\" (UID: \"1fc2c61e-6d9a-4c11-b942-770a3d15153c\") " pod="openshift-insights/insights-runtime-extractor-swzld" Jul 18 00:41:32.006581 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:32.006553 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fc2c61e-6d9a-4c11-b942-770a3d15153c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-swzld\" (UID: \"1fc2c61e-6d9a-4c11-b942-770a3d15153c\") " pod="openshift-insights/insights-runtime-extractor-swzld" Jul 18 00:41:32.007050 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:32.007030 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fc2c61e-6d9a-4c11-b942-770a3d15153c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-swzld\" (UID: \"1fc2c61e-6d9a-4c11-b942-770a3d15153c\") " pod="openshift-insights/insights-runtime-extractor-swzld" Jul 18 00:41:32.008948 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:32.008920 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fc2c61e-6d9a-4c11-b942-770a3d15153c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-swzld\" (UID: \"1fc2c61e-6d9a-4c11-b942-770a3d15153c\") " pod="openshift-insights/insights-runtime-extractor-swzld" Jul 18 00:41:32.025584 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:32.025564 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-swzld" Jul 18 00:41:32.178652 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:32.178480 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dvbgz"] Jul 18 00:41:32.181633 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:41:32.181602 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7471583_1515_4d25_afeb_539e6484f43d.slice/crio-75164644e6bd8eba38a916bdc88cdda0ccfccc708b22ccb476a8ed0cc4856d5d WatchSource:0}: Error finding container 75164644e6bd8eba38a916bdc88cdda0ccfccc708b22ccb476a8ed0cc4856d5d: Status 404 returned error can't find the container with id 75164644e6bd8eba38a916bdc88cdda0ccfccc708b22ccb476a8ed0cc4856d5d Jul 18 00:41:32.202180 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:32.202135 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-swzld"] Jul 18 00:41:32.205375 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:41:32.205351 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fc2c61e_6d9a_4c11_b942_770a3d15153c.slice/crio-43ce027f270577b7489edef2d6cc3c9aaca6006ae5b5dcc08236c781fcbfd85a WatchSource:0}: Error finding container 43ce027f270577b7489edef2d6cc3c9aaca6006ae5b5dcc08236c781fcbfd85a: Status 404 returned error can't find the container with id 43ce027f270577b7489edef2d6cc3c9aaca6006ae5b5dcc08236c781fcbfd85a Jul 18 00:41:32.525001 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:32.524969 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-swzld" event={"ID":"1fc2c61e-6d9a-4c11-b942-770a3d15153c","Type":"ContainerStarted","Data":"43ce027f270577b7489edef2d6cc3c9aaca6006ae5b5dcc08236c781fcbfd85a"} Jul 18 00:41:32.526073 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:32.526048 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dvbgz" event={"ID":"e7471583-1515-4d25-afeb-539e6484f43d","Type":"ContainerStarted","Data":"75164644e6bd8eba38a916bdc88cdda0ccfccc708b22ccb476a8ed0cc4856d5d"} Jul 18 00:41:34.533554 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:34.533518 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dvbgz" event={"ID":"e7471583-1515-4d25-afeb-539e6484f43d","Type":"ContainerStarted","Data":"fedb55a14c498ff3ddf93d2e00f7b5503bea84a6cca9666830f7fe13731fe36d"} Jul 18 00:41:34.533954 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:34.533583 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dvbgz" event={"ID":"e7471583-1515-4d25-afeb-539e6484f43d","Type":"ContainerStarted","Data":"e9c0df6fb2f913a023f3d2dc74b2391d0431f9c692c5f6c6711c5e57497c0d7f"} Jul 18 00:41:34.533954 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:34.533662 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dvbgz" Jul 18 00:41:34.550203 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:34.550154 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dvbgz" podStartSLOduration=16.569796257 podStartE2EDuration="18.5501392s" totalImagesPullingTime="1.980342943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:41:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:34.548958271 +0000 UTC m=+54.900808618" watchObservedRunningTime="2026-07-18 00:41:34.5501392 +0000 UTC m=+54.901989544" Jul 18 00:41:37.544122 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:37.544091 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-swzld" event={"ID":"1fc2c61e-6d9a-4c11-b942-770a3d15153c","Type":"ContainerStarted","Data":"251bfbadc161f89cdba9c43981cfff8de03223245ee112a31461246f2b4d6bb2"} Jul 18 00:41:39.481615 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.481581 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7ddb64499c-fkprz"] Jul 18 00:41:39.577217 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.577180 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7ddb64499c-fkprz"] Jul 18 00:41:39.577365 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.577319 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7ddb64499c-fkprz" Jul 18 00:41:39.579986 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.579958 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-h4ffv\"" Jul 18 00:41:39.580161 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.580094 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 18 00:41:39.580161 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.579958 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 18 00:41:39.580947 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.580925 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 18 00:41:39.586937 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.586914 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 18 00:41:39.672255 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.672224 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a2502e12-3e9c-4a20-a568-719afe10358e-trusted-ca\") pod \"image-registry-7ddb64499c-fkprz\" (UID: \"a2502e12-3e9c-4a20-a568-719afe10358e\") " pod="openshift-image-registry/image-registry-7ddb64499c-fkprz" Jul 18 00:41:39.672451 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.672307 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a2502e12-3e9c-4a20-a568-719afe10358e-registry-tls\") pod \"image-registry-7ddb64499c-fkprz\" (UID: \"a2502e12-3e9c-4a20-a568-719afe10358e\") " pod="openshift-image-registry/image-registry-7ddb64499c-fkprz" Jul 18 00:41:39.672451 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.672330 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a2502e12-3e9c-4a20-a568-719afe10358e-bound-sa-token\") pod \"image-registry-7ddb64499c-fkprz\" (UID: \"a2502e12-3e9c-4a20-a568-719afe10358e\") " pod="openshift-image-registry/image-registry-7ddb64499c-fkprz" Jul 18 00:41:39.672451 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.672356 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qljkv\" (UniqueName: \"kubernetes.io/projected/a2502e12-3e9c-4a20-a568-719afe10358e-kube-api-access-qljkv\") pod \"image-registry-7ddb64499c-fkprz\" (UID: \"a2502e12-3e9c-4a20-a568-719afe10358e\") " pod="openshift-image-registry/image-registry-7ddb64499c-fkprz" Jul 18 00:41:39.672451 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.672394 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a2502e12-3e9c-4a20-a568-719afe10358e-installation-pull-secrets\") pod \"image-registry-7ddb64499c-fkprz\" (UID: \"a2502e12-3e9c-4a20-a568-719afe10358e\") " pod="openshift-image-registry/image-registry-7ddb64499c-fkprz" Jul 18 00:41:39.672451 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.672425 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a2502e12-3e9c-4a20-a568-719afe10358e-ca-trust-extracted\") pod \"image-registry-7ddb64499c-fkprz\" (UID: \"a2502e12-3e9c-4a20-a568-719afe10358e\") " pod="openshift-image-registry/image-registry-7ddb64499c-fkprz" Jul 18 00:41:39.672706 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.672470 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a2502e12-3e9c-4a20-a568-719afe10358e-image-registry-private-configuration\") pod \"image-registry-7ddb64499c-fkprz\" (UID: \"a2502e12-3e9c-4a20-a568-719afe10358e\") " pod="openshift-image-registry/image-registry-7ddb64499c-fkprz" Jul 18 00:41:39.672706 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.672497 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a2502e12-3e9c-4a20-a568-719afe10358e-registry-certificates\") pod \"image-registry-7ddb64499c-fkprz\" (UID: \"a2502e12-3e9c-4a20-a568-719afe10358e\") " pod="openshift-image-registry/image-registry-7ddb64499c-fkprz" Jul 18 00:41:39.773770 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.773039 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a2502e12-3e9c-4a20-a568-719afe10358e-image-registry-private-configuration\") pod \"image-registry-7ddb64499c-fkprz\" (UID: \"a2502e12-3e9c-4a20-a568-719afe10358e\") " pod="openshift-image-registry/image-registry-7ddb64499c-fkprz" Jul 18 00:41:39.773770 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.773100 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a2502e12-3e9c-4a20-a568-719afe10358e-registry-certificates\") pod \"image-registry-7ddb64499c-fkprz\" (UID: \"a2502e12-3e9c-4a20-a568-719afe10358e\") " pod="openshift-image-registry/image-registry-7ddb64499c-fkprz" Jul 18 00:41:39.773770 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.773135 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a2502e12-3e9c-4a20-a568-719afe10358e-trusted-ca\") pod \"image-registry-7ddb64499c-fkprz\" (UID: \"a2502e12-3e9c-4a20-a568-719afe10358e\") " pod="openshift-image-registry/image-registry-7ddb64499c-fkprz" Jul 18 00:41:39.773770 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.773204 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a2502e12-3e9c-4a20-a568-719afe10358e-registry-tls\") pod \"image-registry-7ddb64499c-fkprz\" (UID: \"a2502e12-3e9c-4a20-a568-719afe10358e\") " pod="openshift-image-registry/image-registry-7ddb64499c-fkprz" Jul 18 00:41:39.773770 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.773227 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a2502e12-3e9c-4a20-a568-719afe10358e-bound-sa-token\") pod \"image-registry-7ddb64499c-fkprz\" (UID: \"a2502e12-3e9c-4a20-a568-719afe10358e\") " pod="openshift-image-registry/image-registry-7ddb64499c-fkprz" Jul 18 00:41:39.773770 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.773252 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qljkv\" (UniqueName: \"kubernetes.io/projected/a2502e12-3e9c-4a20-a568-719afe10358e-kube-api-access-qljkv\") pod \"image-registry-7ddb64499c-fkprz\" (UID: \"a2502e12-3e9c-4a20-a568-719afe10358e\") " pod="openshift-image-registry/image-registry-7ddb64499c-fkprz" Jul 18 00:41:39.773770 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.773277 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a2502e12-3e9c-4a20-a568-719afe10358e-installation-pull-secrets\") pod \"image-registry-7ddb64499c-fkprz\" (UID: \"a2502e12-3e9c-4a20-a568-719afe10358e\") " pod="openshift-image-registry/image-registry-7ddb64499c-fkprz" Jul 18 00:41:39.773770 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.773301 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a2502e12-3e9c-4a20-a568-719afe10358e-ca-trust-extracted\") pod \"image-registry-7ddb64499c-fkprz\" (UID: \"a2502e12-3e9c-4a20-a568-719afe10358e\") " pod="openshift-image-registry/image-registry-7ddb64499c-fkprz" Jul 18 00:41:39.773770 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.773609 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a2502e12-3e9c-4a20-a568-719afe10358e-ca-trust-extracted\") pod \"image-registry-7ddb64499c-fkprz\" (UID: \"a2502e12-3e9c-4a20-a568-719afe10358e\") " pod="openshift-image-registry/image-registry-7ddb64499c-fkprz" Jul 18 00:41:39.774233 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.774020 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a2502e12-3e9c-4a20-a568-719afe10358e-registry-certificates\") pod \"image-registry-7ddb64499c-fkprz\" (UID: \"a2502e12-3e9c-4a20-a568-719afe10358e\") " pod="openshift-image-registry/image-registry-7ddb64499c-fkprz" Jul 18 00:41:39.775130 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.775098 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a2502e12-3e9c-4a20-a568-719afe10358e-trusted-ca\") pod \"image-registry-7ddb64499c-fkprz\" (UID: \"a2502e12-3e9c-4a20-a568-719afe10358e\") " pod="openshift-image-registry/image-registry-7ddb64499c-fkprz" Jul 18 00:41:39.776110 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.776080 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a2502e12-3e9c-4a20-a568-719afe10358e-image-registry-private-configuration\") pod \"image-registry-7ddb64499c-fkprz\" (UID: \"a2502e12-3e9c-4a20-a568-719afe10358e\") " pod="openshift-image-registry/image-registry-7ddb64499c-fkprz" Jul 18 00:41:39.776530 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.776508 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a2502e12-3e9c-4a20-a568-719afe10358e-installation-pull-secrets\") pod \"image-registry-7ddb64499c-fkprz\" (UID: \"a2502e12-3e9c-4a20-a568-719afe10358e\") " pod="openshift-image-registry/image-registry-7ddb64499c-fkprz" Jul 18 00:41:39.776883 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.776855 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a2502e12-3e9c-4a20-a568-719afe10358e-registry-tls\") pod \"image-registry-7ddb64499c-fkprz\" (UID: \"a2502e12-3e9c-4a20-a568-719afe10358e\") " pod="openshift-image-registry/image-registry-7ddb64499c-fkprz" Jul 18 00:41:39.782446 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.782393 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qljkv\" (UniqueName: \"kubernetes.io/projected/a2502e12-3e9c-4a20-a568-719afe10358e-kube-api-access-qljkv\") pod \"image-registry-7ddb64499c-fkprz\" (UID: \"a2502e12-3e9c-4a20-a568-719afe10358e\") " pod="openshift-image-registry/image-registry-7ddb64499c-fkprz" Jul 18 00:41:39.782838 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.782814 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a2502e12-3e9c-4a20-a568-719afe10358e-bound-sa-token\") pod \"image-registry-7ddb64499c-fkprz\" (UID: \"a2502e12-3e9c-4a20-a568-719afe10358e\") " pod="openshift-image-registry/image-registry-7ddb64499c-fkprz" Jul 18 00:41:39.890122 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:39.890077 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7ddb64499c-fkprz" Jul 18 00:41:41.093486 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:41.093447 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7ddb64499c-fkprz"] Jul 18 00:41:41.096387 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:41:41.096351 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2502e12_3e9c_4a20_a568_719afe10358e.slice/crio-67b75eda3f0e2554c3ff57701469c0028752ed57958e0e4959d0e5e03c9726d5 WatchSource:0}: Error finding container 67b75eda3f0e2554c3ff57701469c0028752ed57958e0e4959d0e5e03c9726d5: Status 404 returned error can't find the container with id 67b75eda3f0e2554c3ff57701469c0028752ed57958e0e4959d0e5e03c9726d5 Jul 18 00:41:41.560350 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:41.560309 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-swzld" event={"ID":"1fc2c61e-6d9a-4c11-b942-770a3d15153c","Type":"ContainerStarted","Data":"82f905cb46201c1469f9c4319e9faa81d4aaa1cadd726fa4dac0e883ce635ecd"} Jul 18 00:41:41.562419 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:41.562392 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7ddb64499c-fkprz" event={"ID":"a2502e12-3e9c-4a20-a568-719afe10358e","Type":"ContainerStarted","Data":"64939c3fdf2830f060367cd226911ce4bf9734fb830a1bcf071eda4e0d9e1f62"} Jul 18 00:41:41.562419 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:41.562431 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7ddb64499c-fkprz" event={"ID":"a2502e12-3e9c-4a20-a568-719afe10358e","Type":"ContainerStarted","Data":"67b75eda3f0e2554c3ff57701469c0028752ed57958e0e4959d0e5e03c9726d5"} Jul 18 00:41:41.562723 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:41.562708 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7ddb64499c-fkprz" Jul 18 00:41:41.595257 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:41.595194 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7ddb64499c-fkprz" podStartSLOduration=2.59517138 podStartE2EDuration="2.59517138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:41.594603697 +0000 UTC m=+61.946454042" watchObservedRunningTime="2026-07-18 00:41:41.59517138 +0000 UTC m=+61.947021727" Jul 18 00:41:43.466273 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:43.466233 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cwk9l" Jul 18 00:41:44.539057 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:44.539029 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dvbgz" Jul 18 00:41:44.575335 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:44.575289 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-swzld" event={"ID":"1fc2c61e-6d9a-4c11-b942-770a3d15153c","Type":"ContainerStarted","Data":"8813509eb70e6e781404131d5c8c66ef5903e8411ec7fbbe21bd5aa4fd3d7194"} Jul 18 00:41:44.591614 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:44.591558 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-swzld" podStartSLOduration=16.471866688 podStartE2EDuration="28.591541963s" totalImagesPullingTime="12.119675275s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:41:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:44.59062743 +0000 UTC m=+64.942477785" watchObservedRunningTime="2026-07-18 00:41:44.591541963 +0000 UTC m=+64.943392308" Jul 18 00:41:45.826563 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:45.826520 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e616fcde-1cc6-4b83-b568-b0ac7076a7b3-metrics-certs\") pod \"network-metrics-daemon-kj5ss\" (UID: \"e616fcde-1cc6-4b83-b568-b0ac7076a7b3\") " pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:41:45.829852 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:45.829832 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 18 00:41:45.839320 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:45.839295 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e616fcde-1cc6-4b83-b568-b0ac7076a7b3-metrics-certs\") pod \"network-metrics-daemon-kj5ss\" (UID: \"e616fcde-1cc6-4b83-b568-b0ac7076a7b3\") " pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:41:45.927871 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:45.927832 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cb92a57-b715-4664-8342-fe08ec474cc1-original-pull-secret\") pod \"global-pull-secret-syncer-xltqc\" (UID: \"3cb92a57-b715-4664-8342-fe08ec474cc1\") " pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:41:45.928077 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:45.927887 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkgzs\" (UniqueName: \"kubernetes.io/projected/52500c71-f212-4425-a056-470bad08263d-kube-api-access-hkgzs\") pod \"network-check-target-f5gjj\" (UID: \"52500c71-f212-4425-a056-470bad08263d\") " pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:41:45.930381 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:45.930363 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 18 00:41:45.930469 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:45.930450 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 18 00:41:45.940445 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:45.940420 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 18 00:41:45.940612 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:45.940594 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cb92a57-b715-4664-8342-fe08ec474cc1-original-pull-secret\") pod \"global-pull-secret-syncer-xltqc\" (UID: \"3cb92a57-b715-4664-8342-fe08ec474cc1\") " pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:41:45.951745 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:45.951720 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkgzs\" (UniqueName: \"kubernetes.io/projected/52500c71-f212-4425-a056-470bad08263d-kube-api-access-hkgzs\") pod \"network-check-target-f5gjj\" (UID: \"52500c71-f212-4425-a056-470bad08263d\") " pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:41:45.954560 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:45.954539 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d2hq6\"" Jul 18 00:41:45.962495 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:45.962475 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kj5ss" Jul 18 00:41:45.962784 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:45.962768 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-475p8\"" Jul 18 00:41:45.965652 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:45.965637 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xltqc" Jul 18 00:41:45.970779 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:45.970764 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:41:46.140148 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:46.140119 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kj5ss"] Jul 18 00:41:46.143641 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:41:46.143586 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode616fcde_1cc6_4b83_b568_b0ac7076a7b3.slice/crio-43fb21dc4ecfa833be24778db41b40b8e62296914d39912f188440da7cbe050a WatchSource:0}: Error finding container 43fb21dc4ecfa833be24778db41b40b8e62296914d39912f188440da7cbe050a: Status 404 returned error can't find the container with id 43fb21dc4ecfa833be24778db41b40b8e62296914d39912f188440da7cbe050a Jul 18 00:41:46.162776 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:46.162735 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xltqc"] Jul 18 00:41:46.166545 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:41:46.166514 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb92a57_b715_4664_8342_fe08ec474cc1.slice/crio-31d22ae50fbe00ff05c98786afdbb8e6e2038ecf1dfee6ac5129c1a73c602f15 WatchSource:0}: Error finding container 31d22ae50fbe00ff05c98786afdbb8e6e2038ecf1dfee6ac5129c1a73c602f15: Status 404 returned error can't find the container with id 31d22ae50fbe00ff05c98786afdbb8e6e2038ecf1dfee6ac5129c1a73c602f15 Jul 18 00:41:46.188018 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:46.187987 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f5gjj"] Jul 18 00:41:46.190749 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:41:46.190726 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52500c71_f212_4425_a056_470bad08263d.slice/crio-404ccba83515ced5729249dfb19f37f690dac01b2c030ecb8091a5f92e73efcb WatchSource:0}: Error finding container 404ccba83515ced5729249dfb19f37f690dac01b2c030ecb8091a5f92e73efcb: Status 404 returned error can't find the container with id 404ccba83515ced5729249dfb19f37f690dac01b2c030ecb8091a5f92e73efcb Jul 18 00:41:46.583436 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:46.583400 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kj5ss" event={"ID":"e616fcde-1cc6-4b83-b568-b0ac7076a7b3","Type":"ContainerStarted","Data":"43fb21dc4ecfa833be24778db41b40b8e62296914d39912f188440da7cbe050a"} Jul 18 00:41:46.584820 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:46.584765 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f5gjj" event={"ID":"52500c71-f212-4425-a056-470bad08263d","Type":"ContainerStarted","Data":"404ccba83515ced5729249dfb19f37f690dac01b2c030ecb8091a5f92e73efcb"} Jul 18 00:41:46.586087 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:46.586060 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xltqc" event={"ID":"3cb92a57-b715-4664-8342-fe08ec474cc1","Type":"ContainerStarted","Data":"31d22ae50fbe00ff05c98786afdbb8e6e2038ecf1dfee6ac5129c1a73c602f15"} Jul 18 00:41:47.592389 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:47.592289 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kj5ss" event={"ID":"e616fcde-1cc6-4b83-b568-b0ac7076a7b3","Type":"ContainerStarted","Data":"72e59139685fbe5b41be87e049743ada9ab2fb10a30ba8c2b67ca67a6335b071"} Jul 18 00:41:47.592389 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:47.592368 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kj5ss" event={"ID":"e616fcde-1cc6-4b83-b568-b0ac7076a7b3","Type":"ContainerStarted","Data":"821dad64800627718b88a04d87d498bd4b1e84f9351345323a3197b7286f11e3"} Jul 18 00:41:47.610099 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:47.610034 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kj5ss" podStartSLOduration=66.445233236 podStartE2EDuration="1m7.610013508s" totalImagesPullingTime="1.164780272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:47.609420782 +0000 UTC m=+67.961271129" watchObservedRunningTime="2026-07-18 00:41:47.610013508 +0000 UTC m=+67.961863854" Jul 18 00:41:51.609716 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:51.609656 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xltqc" event={"ID":"3cb92a57-b715-4664-8342-fe08ec474cc1","Type":"ContainerStarted","Data":"cbec125ad9d9b8a2f867258d276dbcc4823205afbbfe4dc8aa485463c4a36da7"} Jul 18 00:41:51.611197 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:51.611170 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f5gjj" event={"ID":"52500c71-f212-4425-a056-470bad08263d","Type":"ContainerStarted","Data":"2671eff40d2ee1c42849585f66cb34866950b7ed02ce4e48c2a87300af999bc9"} Jul 18 00:41:51.611331 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:51.611304 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:41:51.623846 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:51.623793 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xltqc" podStartSLOduration=66.064170791 podStartE2EDuration="1m10.623776157s" totalImagesPullingTime="4.559605366s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 00:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:51.623490967 +0000 UTC m=+71.975341315" watchObservedRunningTime="2026-07-18 00:41:51.623776157 +0000 UTC m=+71.975626899" Jul 18 00:41:51.640211 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:41:51.640139 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-f5gjj" podStartSLOduration=67.109939683 podStartE2EDuration="1m11.640123471s" totalImagesPullingTime="4.530183788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:41:51.639251353 +0000 UTC m=+71.991101699" watchObservedRunningTime="2026-07-18 00:41:51.640123471 +0000 UTC m=+71.991973816" Jul 18 00:42:02.571027 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:02.570899 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7ddb64499c-fkprz" Jul 18 00:42:02.633287 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:02.633245 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v2nl7"] Jul 18 00:42:02.635950 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:02.635929 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v2nl7" Jul 18 00:42:02.638352 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:02.638327 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 18 00:42:02.638476 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:02.638400 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-xdxjb\"" Jul 18 00:42:02.647257 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:02.647230 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v2nl7"] Jul 18 00:42:02.754907 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:02.754867 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/522140e6-dd37-4a6f-8c78-e95949b15f4b-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-v2nl7\" (UID: \"522140e6-dd37-4a6f-8c78-e95949b15f4b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v2nl7" Jul 18 00:42:02.855819 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:02.855721 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/522140e6-dd37-4a6f-8c78-e95949b15f4b-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-v2nl7\" (UID: \"522140e6-dd37-4a6f-8c78-e95949b15f4b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v2nl7" Jul 18 00:42:02.856002 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:42:02.855895 2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 18 00:42:02.856002 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:42:02.855992 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/522140e6-dd37-4a6f-8c78-e95949b15f4b-tls-certificates podName:522140e6-dd37-4a6f-8c78-e95949b15f4b nodeName:}" failed. No retries permitted until 2026-07-18 00:42:03.355969252 +0000 UTC m=+83.707819577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/522140e6-dd37-4a6f-8c78-e95949b15f4b-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-v2nl7" (UID: "522140e6-dd37-4a6f-8c78-e95949b15f4b") : secret "prometheus-operator-admission-webhook-tls" not found Jul 18 00:42:03.360426 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:03.360390 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/522140e6-dd37-4a6f-8c78-e95949b15f4b-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-v2nl7\" (UID: \"522140e6-dd37-4a6f-8c78-e95949b15f4b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v2nl7" Jul 18 00:42:03.362977 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:03.362942 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/522140e6-dd37-4a6f-8c78-e95949b15f4b-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-v2nl7\" (UID: \"522140e6-dd37-4a6f-8c78-e95949b15f4b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v2nl7" Jul 18 00:42:03.548651 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:03.548600 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v2nl7" Jul 18 00:42:03.673999 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:03.673966 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v2nl7"] Jul 18 00:42:03.677131 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:42:03.677103 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522140e6_dd37_4a6f_8c78_e95949b15f4b.slice/crio-9443eb3753e1571d481b92cf413647a715a4ac1cf1386244adda294ed3fdd12a WatchSource:0}: Error finding container 9443eb3753e1571d481b92cf413647a715a4ac1cf1386244adda294ed3fdd12a: Status 404 returned error can't find the container with id 9443eb3753e1571d481b92cf413647a715a4ac1cf1386244adda294ed3fdd12a Jul 18 00:42:04.658703 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:04.658645 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v2nl7" event={"ID":"522140e6-dd37-4a6f-8c78-e95949b15f4b","Type":"ContainerStarted","Data":"9443eb3753e1571d481b92cf413647a715a4ac1cf1386244adda294ed3fdd12a"} Jul 18 00:42:05.663180 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:05.663137 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v2nl7" event={"ID":"522140e6-dd37-4a6f-8c78-e95949b15f4b","Type":"ContainerStarted","Data":"f29e8052d8462cb52044179ac583ff9562923001c1be280de72e3611e5189a76"} Jul 18 00:42:05.663630 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:05.663349 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v2nl7" Jul 18 00:42:05.668096 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:05.668076 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v2nl7" Jul 18 00:42:05.682530 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:05.682483 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v2nl7" podStartSLOduration=2.560343507 podStartE2EDuration="3.682469582s" totalImagesPullingTime="1.122126075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:42:05.682095467 +0000 UTC m=+86.033945836" watchObservedRunningTime="2026-07-18 00:42:05.682469582 +0000 UTC m=+86.034319927" Jul 18 00:42:06.684166 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:06.684132 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-xl64x"] Jul 18 00:42:06.687239 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:06.687221 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-xl64x" Jul 18 00:42:06.695438 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:06.695404 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 18 00:42:06.695645 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:06.695438 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 18 00:42:06.695645 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:06.695458 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-l2jmh\"" Jul 18 00:42:06.695842 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:06.695496 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 18 00:42:06.695842 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:06.695530 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 18 00:42:06.695842 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:06.695550 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 18 00:42:06.705082 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:06.705057 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-xl64x"] Jul 18 00:42:06.785253 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:06.785212 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4eefbcad-4399-4d17-9d1a-acd7348c2380-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-xl64x\" (UID: \"4eefbcad-4399-4d17-9d1a-acd7348c2380\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-xl64x" Jul 18 00:42:06.785253 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:06.785248 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4eefbcad-4399-4d17-9d1a-acd7348c2380-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-xl64x\" (UID: \"4eefbcad-4399-4d17-9d1a-acd7348c2380\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-xl64x" Jul 18 00:42:06.785480 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:06.785280 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkxbq\" (UniqueName: \"kubernetes.io/projected/4eefbcad-4399-4d17-9d1a-acd7348c2380-kube-api-access-nkxbq\") pod \"prometheus-operator-5945b5cfd-xl64x\" (UID: \"4eefbcad-4399-4d17-9d1a-acd7348c2380\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-xl64x" Jul 18 00:42:06.785480 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:06.785296 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4eefbcad-4399-4d17-9d1a-acd7348c2380-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-xl64x\" (UID: \"4eefbcad-4399-4d17-9d1a-acd7348c2380\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-xl64x" Jul 18 00:42:06.886204 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:06.886158 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkxbq\" (UniqueName: \"kubernetes.io/projected/4eefbcad-4399-4d17-9d1a-acd7348c2380-kube-api-access-nkxbq\") pod \"prometheus-operator-5945b5cfd-xl64x\" (UID: \"4eefbcad-4399-4d17-9d1a-acd7348c2380\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-xl64x" Jul 18 00:42:06.886204 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:06.886209 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4eefbcad-4399-4d17-9d1a-acd7348c2380-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-xl64x\" (UID: \"4eefbcad-4399-4d17-9d1a-acd7348c2380\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-xl64x" Jul 18 00:42:06.886414 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:06.886273 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4eefbcad-4399-4d17-9d1a-acd7348c2380-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-xl64x\" (UID: \"4eefbcad-4399-4d17-9d1a-acd7348c2380\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-xl64x" Jul 18 00:42:06.886414 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:42:06.886375 2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 18 00:42:06.886414 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:06.886405 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4eefbcad-4399-4d17-9d1a-acd7348c2380-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-xl64x\" (UID: \"4eefbcad-4399-4d17-9d1a-acd7348c2380\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-xl64x" Jul 18 00:42:06.886538 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:42:06.886447 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eefbcad-4399-4d17-9d1a-acd7348c2380-prometheus-operator-tls podName:4eefbcad-4399-4d17-9d1a-acd7348c2380 nodeName:}" failed. No retries permitted until 2026-07-18 00:42:07.38642761 +0000 UTC m=+87.738277934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/4eefbcad-4399-4d17-9d1a-acd7348c2380-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-xl64x" (UID: "4eefbcad-4399-4d17-9d1a-acd7348c2380") : secret "prometheus-operator-tls" not found Jul 18 00:42:06.887071 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:06.887049 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4eefbcad-4399-4d17-9d1a-acd7348c2380-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-xl64x\" (UID: \"4eefbcad-4399-4d17-9d1a-acd7348c2380\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-xl64x" Jul 18 00:42:06.888876 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:06.888855 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4eefbcad-4399-4d17-9d1a-acd7348c2380-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-xl64x\" (UID: \"4eefbcad-4399-4d17-9d1a-acd7348c2380\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-xl64x" Jul 18 00:42:06.895839 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:06.895812 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkxbq\" (UniqueName: \"kubernetes.io/projected/4eefbcad-4399-4d17-9d1a-acd7348c2380-kube-api-access-nkxbq\") pod \"prometheus-operator-5945b5cfd-xl64x\" (UID: \"4eefbcad-4399-4d17-9d1a-acd7348c2380\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-xl64x" Jul 18 00:42:07.389093 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:07.389051 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4eefbcad-4399-4d17-9d1a-acd7348c2380-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-xl64x\" (UID: \"4eefbcad-4399-4d17-9d1a-acd7348c2380\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-xl64x" Jul 18 00:42:07.391763 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:07.391728 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4eefbcad-4399-4d17-9d1a-acd7348c2380-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-xl64x\" (UID: \"4eefbcad-4399-4d17-9d1a-acd7348c2380\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-xl64x" Jul 18 00:42:07.601211 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:07.601168 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-xl64x" Jul 18 00:42:07.741218 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:07.741177 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-xl64x"] Jul 18 00:42:07.744109 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:42:07.744083 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eefbcad_4399_4d17_9d1a_acd7348c2380.slice/crio-ad9f183859bed275c064469772ec12c101b2305c736dde054b6d230082c56956 WatchSource:0}: Error finding container ad9f183859bed275c064469772ec12c101b2305c736dde054b6d230082c56956: Status 404 returned error can't find the container with id ad9f183859bed275c064469772ec12c101b2305c736dde054b6d230082c56956 Jul 18 00:42:08.675297 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:08.675251 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-xl64x" event={"ID":"4eefbcad-4399-4d17-9d1a-acd7348c2380","Type":"ContainerStarted","Data":"ad9f183859bed275c064469772ec12c101b2305c736dde054b6d230082c56956"} Jul 18 00:42:09.680440 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:09.680406 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-xl64x" event={"ID":"4eefbcad-4399-4d17-9d1a-acd7348c2380","Type":"ContainerStarted","Data":"ea2586d08b7c926f51096e08ff003df8b632a8af98d8d3abc96843e7ecf775de"} Jul 18 00:42:09.680852 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:09.680459 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-xl64x" event={"ID":"4eefbcad-4399-4d17-9d1a-acd7348c2380","Type":"ContainerStarted","Data":"1575b97f60b3b5796e212640810db962a146f84acf6594c4e9a34ac8c119d77f"} Jul 18 00:42:09.697660 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:09.697609 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-xl64x" podStartSLOduration=2.595541461 podStartE2EDuration="3.697590695s" totalImagesPullingTime="1.102049234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:42:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:42:09.697278381 +0000 UTC m=+90.049128726" watchObservedRunningTime="2026-07-18 00:42:09.697590695 +0000 UTC m=+90.049441045" Jul 18 00:42:10.339070 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:10.339035 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f855dfbd7-nzqwb"] Jul 18 00:42:10.347994 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:10.347970 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f855dfbd7-nzqwb" Jul 18 00:42:10.350744 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:10.350717 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 18 00:42:10.350947 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:10.350930 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 18 00:42:10.351002 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:10.350959 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 18 00:42:10.351048 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:10.350998 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 18 00:42:10.352111 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:10.352090 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-zcw6x\"" Jul 18 00:42:10.352212 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:10.352108 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 18 00:42:10.352628 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:10.352603 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 18 00:42:10.352942 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:10.352920 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 18 00:42:10.359396 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:10.359372 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f855dfbd7-nzqwb"] Jul 18 00:42:10.512501 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:10.512465 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc655\" (UniqueName: \"kubernetes.io/projected/a20006f4-3771-49c5-9516-cb23ad68fcc3-kube-api-access-gc655\") pod \"console-6f855dfbd7-nzqwb\" (UID: \"a20006f4-3771-49c5-9516-cb23ad68fcc3\") " pod="openshift-console/console-6f855dfbd7-nzqwb" Jul 18 00:42:10.512732 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:10.512517 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a20006f4-3771-49c5-9516-cb23ad68fcc3-console-oauth-config\") pod \"console-6f855dfbd7-nzqwb\" (UID: \"a20006f4-3771-49c5-9516-cb23ad68fcc3\") " pod="openshift-console/console-6f855dfbd7-nzqwb" Jul 18 00:42:10.512732 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:10.512545 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a20006f4-3771-49c5-9516-cb23ad68fcc3-console-config\") pod \"console-6f855dfbd7-nzqwb\" (UID: \"a20006f4-3771-49c5-9516-cb23ad68fcc3\") " pod="openshift-console/console-6f855dfbd7-nzqwb" Jul 18 00:42:10.512732 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:10.512620 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a20006f4-3771-49c5-9516-cb23ad68fcc3-console-serving-cert\") pod \"console-6f855dfbd7-nzqwb\" (UID: \"a20006f4-3771-49c5-9516-cb23ad68fcc3\") " pod="openshift-console/console-6f855dfbd7-nzqwb" Jul 18 00:42:10.512732 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:10.512698 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a20006f4-3771-49c5-9516-cb23ad68fcc3-service-ca\") pod \"console-6f855dfbd7-nzqwb\" (UID: \"a20006f4-3771-49c5-9516-cb23ad68fcc3\") " pod="openshift-console/console-6f855dfbd7-nzqwb" Jul 18 00:42:10.512904 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:10.512744 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a20006f4-3771-49c5-9516-cb23ad68fcc3-oauth-serving-cert\") pod \"console-6f855dfbd7-nzqwb\" (UID: \"a20006f4-3771-49c5-9516-cb23ad68fcc3\") " pod="openshift-console/console-6f855dfbd7-nzqwb" Jul 18 00:42:10.613498 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:10.613404 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc655\" (UniqueName: \"kubernetes.io/projected/a20006f4-3771-49c5-9516-cb23ad68fcc3-kube-api-access-gc655\") pod \"console-6f855dfbd7-nzqwb\" (UID: \"a20006f4-3771-49c5-9516-cb23ad68fcc3\") " pod="openshift-console/console-6f855dfbd7-nzqwb" Jul 18 00:42:10.613498 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:10.613455 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a20006f4-3771-49c5-9516-cb23ad68fcc3-console-oauth-config\") pod \"console-6f855dfbd7-nzqwb\" (UID: \"a20006f4-3771-49c5-9516-cb23ad68fcc3\") " pod="openshift-console/console-6f855dfbd7-nzqwb" Jul 18 00:42:10.613498 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:10.613478 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a20006f4-3771-49c5-9516-cb23ad68fcc3-console-config\") pod \"console-6f855dfbd7-nzqwb\" (UID: \"a20006f4-3771-49c5-9516-cb23ad68fcc3\") " pod="openshift-console/console-6f855dfbd7-nzqwb" Jul 18 00:42:10.613498 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:10.613494 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a20006f4-3771-49c5-9516-cb23ad68fcc3-console-serving-cert\") pod \"console-6f855dfbd7-nzqwb\" (UID: \"a20006f4-3771-49c5-9516-cb23ad68fcc3\") " pod="openshift-console/console-6f855dfbd7-nzqwb" Jul 18 00:42:10.613853 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:10.613516 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a20006f4-3771-49c5-9516-cb23ad68fcc3-service-ca\") pod \"console-6f855dfbd7-nzqwb\" (UID: \"a20006f4-3771-49c5-9516-cb23ad68fcc3\") " pod="openshift-console/console-6f855dfbd7-nzqwb" Jul 18 00:42:10.613853 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:10.613549 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a20006f4-3771-49c5-9516-cb23ad68fcc3-oauth-serving-cert\") pod \"console-6f855dfbd7-nzqwb\" (UID: \"a20006f4-3771-49c5-9516-cb23ad68fcc3\") " pod="openshift-console/console-6f855dfbd7-nzqwb" Jul 18 00:42:10.614284 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:10.614230 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a20006f4-3771-49c5-9516-cb23ad68fcc3-console-config\") pod \"console-6f855dfbd7-nzqwb\" (UID: \"a20006f4-3771-49c5-9516-cb23ad68fcc3\") " pod="openshift-console/console-6f855dfbd7-nzqwb" Jul 18 00:42:10.614284 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:10.614299 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a20006f4-3771-49c5-9516-cb23ad68fcc3-oauth-serving-cert\") pod \"console-6f855dfbd7-nzqwb\" (UID: \"a20006f4-3771-49c5-9516-cb23ad68fcc3\") " pod="openshift-console/console-6f855dfbd7-nzqwb" Jul 18 00:42:10.614552 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:10.614499 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a20006f4-3771-49c5-9516-cb23ad68fcc3-service-ca\") pod \"console-6f855dfbd7-nzqwb\" (UID: \"a20006f4-3771-49c5-9516-cb23ad68fcc3\") " pod="openshift-console/console-6f855dfbd7-nzqwb" Jul 18 00:42:10.616243 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:10.616218 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a20006f4-3771-49c5-9516-cb23ad68fcc3-console-oauth-config\") pod \"console-6f855dfbd7-nzqwb\" (UID: \"a20006f4-3771-49c5-9516-cb23ad68fcc3\") " pod="openshift-console/console-6f855dfbd7-nzqwb" Jul 18 00:42:10.616338 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:10.616323 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a20006f4-3771-49c5-9516-cb23ad68fcc3-console-serving-cert\") pod \"console-6f855dfbd7-nzqwb\" (UID: \"a20006f4-3771-49c5-9516-cb23ad68fcc3\") " pod="openshift-console/console-6f855dfbd7-nzqwb" Jul 18 00:42:10.622213 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:10.622190 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc655\" (UniqueName: \"kubernetes.io/projected/a20006f4-3771-49c5-9516-cb23ad68fcc3-kube-api-access-gc655\") pod \"console-6f855dfbd7-nzqwb\" (UID: \"a20006f4-3771-49c5-9516-cb23ad68fcc3\") " pod="openshift-console/console-6f855dfbd7-nzqwb" Jul 18 00:42:10.660124 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:10.660085 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f855dfbd7-nzqwb" Jul 18 00:42:10.854948 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:10.854915 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f855dfbd7-nzqwb"] Jul 18 00:42:10.857896 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:42:10.857861 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda20006f4_3771_49c5_9516_cb23ad68fcc3.slice/crio-c7ac5147fdec6bcae0f8878b016ac7cce503f977626e520a9bcafe631b509d3e WatchSource:0}: Error finding container c7ac5147fdec6bcae0f8878b016ac7cce503f977626e520a9bcafe631b509d3e: Status 404 returned error can't find the container with id c7ac5147fdec6bcae0f8878b016ac7cce503f977626e520a9bcafe631b509d3e Jul 18 00:42:11.690551 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:11.690509 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f855dfbd7-nzqwb" event={"ID":"a20006f4-3771-49c5-9516-cb23ad68fcc3","Type":"ContainerStarted","Data":"c7ac5147fdec6bcae0f8878b016ac7cce503f977626e520a9bcafe631b509d3e"} Jul 18 00:42:12.079630 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.079593 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8nrwv"] Jul 18 00:42:12.147522 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.147491 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8nrwv" Jul 18 00:42:12.151734 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.151218 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 18 00:42:12.151734 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.151242 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 18 00:42:12.151734 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.151493 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 18 00:42:12.151734 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.151551 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hlh57\"" Jul 18 00:42:12.227253 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.227215 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2892ae85-8898-4539-b4ee-0be960d2da88-node-exporter-wtmp\") pod \"node-exporter-8nrwv\" (UID: \"2892ae85-8898-4539-b4ee-0be960d2da88\") " pod="openshift-monitoring/node-exporter-8nrwv" Jul 18 00:42:12.227428 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.227276 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2892ae85-8898-4539-b4ee-0be960d2da88-sys\") pod \"node-exporter-8nrwv\" (UID: \"2892ae85-8898-4539-b4ee-0be960d2da88\") " pod="openshift-monitoring/node-exporter-8nrwv" Jul 18 00:42:12.227428 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.227313 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2892ae85-8898-4539-b4ee-0be960d2da88-node-exporter-textfile\") pod \"node-exporter-8nrwv\" (UID: \"2892ae85-8898-4539-b4ee-0be960d2da88\") " pod="openshift-monitoring/node-exporter-8nrwv" Jul 18 00:42:12.227428 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.227344 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2892ae85-8898-4539-b4ee-0be960d2da88-node-exporter-accelerators-collector-config\") pod \"node-exporter-8nrwv\" (UID: \"2892ae85-8898-4539-b4ee-0be960d2da88\") " pod="openshift-monitoring/node-exporter-8nrwv" Jul 18 00:42:12.227428 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.227368 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtg8h\" (UniqueName: \"kubernetes.io/projected/2892ae85-8898-4539-b4ee-0be960d2da88-kube-api-access-jtg8h\") pod \"node-exporter-8nrwv\" (UID: \"2892ae85-8898-4539-b4ee-0be960d2da88\") " pod="openshift-monitoring/node-exporter-8nrwv" Jul 18 00:42:12.227428 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.227408 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2892ae85-8898-4539-b4ee-0be960d2da88-root\") pod \"node-exporter-8nrwv\" (UID: \"2892ae85-8898-4539-b4ee-0be960d2da88\") " pod="openshift-monitoring/node-exporter-8nrwv" Jul 18 00:42:12.227718 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.227448 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2892ae85-8898-4539-b4ee-0be960d2da88-metrics-client-ca\") pod \"node-exporter-8nrwv\" (UID: \"2892ae85-8898-4539-b4ee-0be960d2da88\") " pod="openshift-monitoring/node-exporter-8nrwv" Jul 18 00:42:12.227718 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.227474 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2892ae85-8898-4539-b4ee-0be960d2da88-node-exporter-tls\") pod \"node-exporter-8nrwv\" (UID: \"2892ae85-8898-4539-b4ee-0be960d2da88\") " pod="openshift-monitoring/node-exporter-8nrwv" Jul 18 00:42:12.227718 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.227496 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2892ae85-8898-4539-b4ee-0be960d2da88-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8nrwv\" (UID: \"2892ae85-8898-4539-b4ee-0be960d2da88\") " pod="openshift-monitoring/node-exporter-8nrwv" Jul 18 00:42:12.329062 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.328488 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2892ae85-8898-4539-b4ee-0be960d2da88-root\") pod \"node-exporter-8nrwv\" (UID: \"2892ae85-8898-4539-b4ee-0be960d2da88\") " pod="openshift-monitoring/node-exporter-8nrwv" Jul 18 00:42:12.329062 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.328555 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2892ae85-8898-4539-b4ee-0be960d2da88-metrics-client-ca\") pod \"node-exporter-8nrwv\" (UID: \"2892ae85-8898-4539-b4ee-0be960d2da88\") " pod="openshift-monitoring/node-exporter-8nrwv" Jul 18 00:42:12.329062 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.328587 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2892ae85-8898-4539-b4ee-0be960d2da88-root\") pod \"node-exporter-8nrwv\" (UID: \"2892ae85-8898-4539-b4ee-0be960d2da88\") " pod="openshift-monitoring/node-exporter-8nrwv" Jul 18 00:42:12.329062 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.328595 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2892ae85-8898-4539-b4ee-0be960d2da88-node-exporter-tls\") pod \"node-exporter-8nrwv\" (UID: \"2892ae85-8898-4539-b4ee-0be960d2da88\") " pod="openshift-monitoring/node-exporter-8nrwv" Jul 18 00:42:12.329062 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.328651 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2892ae85-8898-4539-b4ee-0be960d2da88-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8nrwv\" (UID: \"2892ae85-8898-4539-b4ee-0be960d2da88\") " pod="openshift-monitoring/node-exporter-8nrwv" Jul 18 00:42:12.329062 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.328720 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2892ae85-8898-4539-b4ee-0be960d2da88-node-exporter-wtmp\") pod \"node-exporter-8nrwv\" (UID: \"2892ae85-8898-4539-b4ee-0be960d2da88\") " pod="openshift-monitoring/node-exporter-8nrwv" Jul 18 00:42:12.329062 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.328761 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2892ae85-8898-4539-b4ee-0be960d2da88-sys\") pod \"node-exporter-8nrwv\" (UID: \"2892ae85-8898-4539-b4ee-0be960d2da88\") " pod="openshift-monitoring/node-exporter-8nrwv" Jul 18 00:42:12.329062 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.328799 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2892ae85-8898-4539-b4ee-0be960d2da88-node-exporter-textfile\") pod \"node-exporter-8nrwv\" (UID: \"2892ae85-8898-4539-b4ee-0be960d2da88\") " pod="openshift-monitoring/node-exporter-8nrwv" Jul 18 00:42:12.329062 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.328821 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2892ae85-8898-4539-b4ee-0be960d2da88-node-exporter-accelerators-collector-config\") pod \"node-exporter-8nrwv\" (UID: \"2892ae85-8898-4539-b4ee-0be960d2da88\") " pod="openshift-monitoring/node-exporter-8nrwv" Jul 18 00:42:12.329062 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.328842 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtg8h\" (UniqueName: \"kubernetes.io/projected/2892ae85-8898-4539-b4ee-0be960d2da88-kube-api-access-jtg8h\") pod \"node-exporter-8nrwv\" (UID: \"2892ae85-8898-4539-b4ee-0be960d2da88\") " pod="openshift-monitoring/node-exporter-8nrwv" Jul 18 00:42:12.329062 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.328879 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2892ae85-8898-4539-b4ee-0be960d2da88-sys\") pod \"node-exporter-8nrwv\" (UID: \"2892ae85-8898-4539-b4ee-0be960d2da88\") " pod="openshift-monitoring/node-exporter-8nrwv" Jul 18 00:42:12.329062 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.328883 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2892ae85-8898-4539-b4ee-0be960d2da88-node-exporter-wtmp\") pod \"node-exporter-8nrwv\" (UID: \"2892ae85-8898-4539-b4ee-0be960d2da88\") " pod="openshift-monitoring/node-exporter-8nrwv" Jul 18 00:42:12.330002 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.329129 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2892ae85-8898-4539-b4ee-0be960d2da88-node-exporter-textfile\") pod \"node-exporter-8nrwv\" (UID: \"2892ae85-8898-4539-b4ee-0be960d2da88\") " pod="openshift-monitoring/node-exporter-8nrwv" Jul 18 00:42:12.330002 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.329229 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2892ae85-8898-4539-b4ee-0be960d2da88-metrics-client-ca\") pod \"node-exporter-8nrwv\" (UID: \"2892ae85-8898-4539-b4ee-0be960d2da88\") " pod="openshift-monitoring/node-exporter-8nrwv" Jul 18 00:42:12.330002 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.329341 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2892ae85-8898-4539-b4ee-0be960d2da88-node-exporter-accelerators-collector-config\") pod \"node-exporter-8nrwv\" (UID: \"2892ae85-8898-4539-b4ee-0be960d2da88\") " pod="openshift-monitoring/node-exporter-8nrwv" Jul 18 00:42:12.331495 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.331474 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2892ae85-8898-4539-b4ee-0be960d2da88-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8nrwv\" (UID: \"2892ae85-8898-4539-b4ee-0be960d2da88\") " pod="openshift-monitoring/node-exporter-8nrwv" Jul 18 00:42:12.332466 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.332444 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2892ae85-8898-4539-b4ee-0be960d2da88-node-exporter-tls\") pod \"node-exporter-8nrwv\" (UID: \"2892ae85-8898-4539-b4ee-0be960d2da88\") " pod="openshift-monitoring/node-exporter-8nrwv" Jul 18 00:42:12.338035 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.338010 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtg8h\" (UniqueName: \"kubernetes.io/projected/2892ae85-8898-4539-b4ee-0be960d2da88-kube-api-access-jtg8h\") pod \"node-exporter-8nrwv\" (UID: \"2892ae85-8898-4539-b4ee-0be960d2da88\") " pod="openshift-monitoring/node-exporter-8nrwv" Jul 18 00:42:12.461789 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.461742 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8nrwv" Jul 18 00:42:12.503188 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:42:12.503156 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2892ae85_8898_4539_b4ee_0be960d2da88.slice/crio-add81fd4729c704edb85c4357c81865e5adf9e914b71e89bcc2b78a2af733df8 WatchSource:0}: Error finding container add81fd4729c704edb85c4357c81865e5adf9e914b71e89bcc2b78a2af733df8: Status 404 returned error can't find the container with id add81fd4729c704edb85c4357c81865e5adf9e914b71e89bcc2b78a2af733df8 Jul 18 00:42:12.696260 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:12.696182 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8nrwv" event={"ID":"2892ae85-8898-4539-b4ee-0be960d2da88","Type":"ContainerStarted","Data":"add81fd4729c704edb85c4357c81865e5adf9e914b71e89bcc2b78a2af733df8"} Jul 18 00:42:14.706157 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:14.706111 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f855dfbd7-nzqwb" event={"ID":"a20006f4-3771-49c5-9516-cb23ad68fcc3","Type":"ContainerStarted","Data":"7a7ce70b026c3776c9482873c884c58b9f8b8428996343a5f0f86506d88f852d"} Jul 18 00:42:14.727754 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:14.727703 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f855dfbd7-nzqwb" podStartSLOduration=1.52192255 podStartE2EDuration="4.727686211s" totalImagesPullingTime="3.205763661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:42:14.726288151 +0000 UTC m=+95.078138496" watchObservedRunningTime="2026-07-18 00:42:14.727686211 +0000 UTC m=+95.079536546" Jul 18 00:42:15.711015 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:15.710974 2580 generic.go:356] "Generic (PLEG): container finished" podID="2892ae85-8898-4539-b4ee-0be960d2da88" containerID="0856084ebb894ede49f4117edb46d6f01d909bf2916a4c663cab61adb892a9a6" exitCode=0 Jul 18 00:42:15.711404 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:15.711068 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8nrwv" event={"ID":"2892ae85-8898-4539-b4ee-0be960d2da88","Type":"ContainerDied","Data":"0856084ebb894ede49f4117edb46d6f01d909bf2916a4c663cab61adb892a9a6"} Jul 18 00:42:16.716606 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:16.716568 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8nrwv" event={"ID":"2892ae85-8898-4539-b4ee-0be960d2da88","Type":"ContainerStarted","Data":"7a92e3cf223118433efc157c27477744b5490f60cae7d684a2a677e34451858f"} Jul 18 00:42:16.717003 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:16.716627 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8nrwv" event={"ID":"2892ae85-8898-4539-b4ee-0be960d2da88","Type":"ContainerStarted","Data":"dc2bb0fa19c42b79928c3a7635168602ab43e9555a14dccb7ab2178646c24cb6"} Jul 18 00:42:19.325205 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:19.325145 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8nrwv" podStartSLOduration=4.804498031 podStartE2EDuration="7.325127775s" totalImagesPullingTime="2.425591805s" totalInitContainerRuntime="95.037939ms" isStatefulPod=true podCreationTimestamp="2026-07-18 00:42:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:42:16.734994729 +0000 UTC m=+97.086845074" watchObservedRunningTime="2026-07-18 00:42:19.325127775 +0000 UTC m=+99.676978120" Jul 18 00:42:19.325700 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:19.325683 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d595d7cbd-lxsw2"] Jul 18 00:42:19.331881 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:19.331864 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d595d7cbd-lxsw2" Jul 18 00:42:19.340205 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:19.340176 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 18 00:42:19.340788 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:19.340765 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d595d7cbd-lxsw2"] Jul 18 00:42:19.488874 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:19.488818 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f188b37-4be7-4fd1-b4c9-65021bba45ca-console-serving-cert\") pod \"console-7d595d7cbd-lxsw2\" (UID: \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\") " pod="openshift-console/console-7d595d7cbd-lxsw2" Jul 18 00:42:19.488874 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:19.488876 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f188b37-4be7-4fd1-b4c9-65021bba45ca-console-config\") pod \"console-7d595d7cbd-lxsw2\" (UID: \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\") " pod="openshift-console/console-7d595d7cbd-lxsw2" Jul 18 00:42:19.489095 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:19.488925 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f188b37-4be7-4fd1-b4c9-65021bba45ca-trusted-ca-bundle\") pod \"console-7d595d7cbd-lxsw2\" (UID: \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\") " pod="openshift-console/console-7d595d7cbd-lxsw2" Jul 18 00:42:19.489095 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:19.488982 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdnhd\" (UniqueName: \"kubernetes.io/projected/3f188b37-4be7-4fd1-b4c9-65021bba45ca-kube-api-access-jdnhd\") pod \"console-7d595d7cbd-lxsw2\" (UID: \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\") " pod="openshift-console/console-7d595d7cbd-lxsw2" Jul 18 00:42:19.489095 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:19.489032 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f188b37-4be7-4fd1-b4c9-65021bba45ca-oauth-serving-cert\") pod \"console-7d595d7cbd-lxsw2\" (UID: \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\") " pod="openshift-console/console-7d595d7cbd-lxsw2" Jul 18 00:42:19.489095 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:19.489080 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f188b37-4be7-4fd1-b4c9-65021bba45ca-service-ca\") pod \"console-7d595d7cbd-lxsw2\" (UID: \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\") " pod="openshift-console/console-7d595d7cbd-lxsw2" Jul 18 00:42:19.489213 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:19.489109 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f188b37-4be7-4fd1-b4c9-65021bba45ca-console-oauth-config\") pod \"console-7d595d7cbd-lxsw2\" (UID: \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\") " pod="openshift-console/console-7d595d7cbd-lxsw2" Jul 18 00:42:19.589993 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:19.589911 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f188b37-4be7-4fd1-b4c9-65021bba45ca-service-ca\") pod \"console-7d595d7cbd-lxsw2\" (UID: \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\") " pod="openshift-console/console-7d595d7cbd-lxsw2" Jul 18 00:42:19.589993 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:19.589958 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f188b37-4be7-4fd1-b4c9-65021bba45ca-console-oauth-config\") pod \"console-7d595d7cbd-lxsw2\" (UID: \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\") " pod="openshift-console/console-7d595d7cbd-lxsw2" Jul 18 00:42:19.590208 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:19.590001 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f188b37-4be7-4fd1-b4c9-65021bba45ca-console-serving-cert\") pod \"console-7d595d7cbd-lxsw2\" (UID: \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\") " pod="openshift-console/console-7d595d7cbd-lxsw2" Jul 18 00:42:19.590208 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:19.590033 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f188b37-4be7-4fd1-b4c9-65021bba45ca-console-config\") pod \"console-7d595d7cbd-lxsw2\" (UID: \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\") " pod="openshift-console/console-7d595d7cbd-lxsw2" Jul 18 00:42:19.590208 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:19.590057 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f188b37-4be7-4fd1-b4c9-65021bba45ca-trusted-ca-bundle\") pod \"console-7d595d7cbd-lxsw2\" (UID: \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\") " pod="openshift-console/console-7d595d7cbd-lxsw2" Jul 18 00:42:19.590208 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:19.590101 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdnhd\" (UniqueName: \"kubernetes.io/projected/3f188b37-4be7-4fd1-b4c9-65021bba45ca-kube-api-access-jdnhd\") pod \"console-7d595d7cbd-lxsw2\" (UID: \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\") " pod="openshift-console/console-7d595d7cbd-lxsw2" Jul 18 00:42:19.590208 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:19.590123 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f188b37-4be7-4fd1-b4c9-65021bba45ca-oauth-serving-cert\") pod \"console-7d595d7cbd-lxsw2\" (UID: \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\") " pod="openshift-console/console-7d595d7cbd-lxsw2" Jul 18 00:42:19.590832 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:19.590804 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f188b37-4be7-4fd1-b4c9-65021bba45ca-service-ca\") pod \"console-7d595d7cbd-lxsw2\" (UID: \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\") " pod="openshift-console/console-7d595d7cbd-lxsw2" Jul 18 00:42:19.590933 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:19.590813 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f188b37-4be7-4fd1-b4c9-65021bba45ca-oauth-serving-cert\") pod \"console-7d595d7cbd-lxsw2\" (UID: \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\") " pod="openshift-console/console-7d595d7cbd-lxsw2" Jul 18 00:42:19.590991 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:19.590929 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f188b37-4be7-4fd1-b4c9-65021bba45ca-trusted-ca-bundle\") pod \"console-7d595d7cbd-lxsw2\" (UID: \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\") " pod="openshift-console/console-7d595d7cbd-lxsw2" Jul 18 00:42:19.591047 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:19.591023 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f188b37-4be7-4fd1-b4c9-65021bba45ca-console-config\") pod \"console-7d595d7cbd-lxsw2\" (UID: \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\") " pod="openshift-console/console-7d595d7cbd-lxsw2" Jul 18 00:42:19.592578 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:19.592554 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f188b37-4be7-4fd1-b4c9-65021bba45ca-console-serving-cert\") pod \"console-7d595d7cbd-lxsw2\" (UID: \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\") " pod="openshift-console/console-7d595d7cbd-lxsw2" Jul 18 00:42:19.592657 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:19.592620 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f188b37-4be7-4fd1-b4c9-65021bba45ca-console-oauth-config\") pod \"console-7d595d7cbd-lxsw2\" (UID: \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\") " pod="openshift-console/console-7d595d7cbd-lxsw2" Jul 18 00:42:19.599079 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:19.599061 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdnhd\" (UniqueName: \"kubernetes.io/projected/3f188b37-4be7-4fd1-b4c9-65021bba45ca-kube-api-access-jdnhd\") pod \"console-7d595d7cbd-lxsw2\" (UID: \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\") " pod="openshift-console/console-7d595d7cbd-lxsw2" Jul 18 00:42:19.642084 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:19.642046 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d595d7cbd-lxsw2" Jul 18 00:42:19.770240 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:19.770206 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d595d7cbd-lxsw2"] Jul 18 00:42:19.773314 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:42:19.773289 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f188b37_4be7_4fd1_b4c9_65021bba45ca.slice/crio-6c0bca712c5e809bd5099603ebb9902854806a938d5738e730df195d9449f17a WatchSource:0}: Error finding container 6c0bca712c5e809bd5099603ebb9902854806a938d5738e730df195d9449f17a: Status 404 returned error can't find the container with id 6c0bca712c5e809bd5099603ebb9902854806a938d5738e730df195d9449f17a Jul 18 00:42:20.661162 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:20.661122 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f855dfbd7-nzqwb" Jul 18 00:42:20.661636 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:20.661183 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f855dfbd7-nzqwb" Jul 18 00:42:20.666235 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:20.666209 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f855dfbd7-nzqwb" Jul 18 00:42:20.735055 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:20.735024 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d595d7cbd-lxsw2" event={"ID":"3f188b37-4be7-4fd1-b4c9-65021bba45ca","Type":"ContainerStarted","Data":"6d15db08956636848874f7568a93a36fdfe84b92aa66213c93fc2c5a0f5e0c7e"} Jul 18 00:42:20.735220 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:20.735078 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d595d7cbd-lxsw2" event={"ID":"3f188b37-4be7-4fd1-b4c9-65021bba45ca","Type":"ContainerStarted","Data":"6c0bca712c5e809bd5099603ebb9902854806a938d5738e730df195d9449f17a"} Jul 18 00:42:20.742805 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:20.739610 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f855dfbd7-nzqwb" Jul 18 00:42:20.753876 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:20.753822 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d595d7cbd-lxsw2" podStartSLOduration=1.75377444 podStartE2EDuration="1.75377444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:42:20.753358299 +0000 UTC m=+101.105208645" watchObservedRunningTime="2026-07-18 00:42:20.75377444 +0000 UTC m=+101.105624784" Jul 18 00:42:22.616363 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:22.616334 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-f5gjj" Jul 18 00:42:29.643294 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:29.643246 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d595d7cbd-lxsw2" Jul 18 00:42:29.643887 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:29.643317 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d595d7cbd-lxsw2" Jul 18 00:42:29.648003 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:29.647980 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d595d7cbd-lxsw2" Jul 18 00:42:29.769142 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:29.769115 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d595d7cbd-lxsw2" Jul 18 00:42:29.810029 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:29.809987 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f855dfbd7-nzqwb"] Jul 18 00:42:54.837414 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:54.837354 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f855dfbd7-nzqwb" podUID="a20006f4-3771-49c5-9516-cb23ad68fcc3" containerName="console" containerID="cri-o://7a7ce70b026c3776c9482873c884c58b9f8b8428996343a5f0f86506d88f852d" gracePeriod=15 Jul 18 00:42:55.079073 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:55.079051 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f855dfbd7-nzqwb" Jul 18 00:42:55.156251 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:55.156154 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a20006f4-3771-49c5-9516-cb23ad68fcc3-console-config\") pod \"a20006f4-3771-49c5-9516-cb23ad68fcc3\" (UID: \"a20006f4-3771-49c5-9516-cb23ad68fcc3\") " Jul 18 00:42:55.156251 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:55.156207 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a20006f4-3771-49c5-9516-cb23ad68fcc3-console-serving-cert\") pod \"a20006f4-3771-49c5-9516-cb23ad68fcc3\" (UID: \"a20006f4-3771-49c5-9516-cb23ad68fcc3\") " Jul 18 00:42:55.156251 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:55.156246 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a20006f4-3771-49c5-9516-cb23ad68fcc3-service-ca\") pod \"a20006f4-3771-49c5-9516-cb23ad68fcc3\" (UID: \"a20006f4-3771-49c5-9516-cb23ad68fcc3\") " Jul 18 00:42:55.156513 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:55.156270 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a20006f4-3771-49c5-9516-cb23ad68fcc3-oauth-serving-cert\") pod \"a20006f4-3771-49c5-9516-cb23ad68fcc3\" (UID: \"a20006f4-3771-49c5-9516-cb23ad68fcc3\") " Jul 18 00:42:55.156513 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:55.156292 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a20006f4-3771-49c5-9516-cb23ad68fcc3-console-oauth-config\") pod \"a20006f4-3771-49c5-9516-cb23ad68fcc3\" (UID: \"a20006f4-3771-49c5-9516-cb23ad68fcc3\") " Jul 18 00:42:55.156513 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:55.156350 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gc655\" (UniqueName: \"kubernetes.io/projected/a20006f4-3771-49c5-9516-cb23ad68fcc3-kube-api-access-gc655\") pod \"a20006f4-3771-49c5-9516-cb23ad68fcc3\" (UID: \"a20006f4-3771-49c5-9516-cb23ad68fcc3\") " Jul 18 00:42:55.156763 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:55.156732 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a20006f4-3771-49c5-9516-cb23ad68fcc3-console-config" (OuterVolumeSpecName: "console-config") pod "a20006f4-3771-49c5-9516-cb23ad68fcc3" (UID: "a20006f4-3771-49c5-9516-cb23ad68fcc3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 00:42:55.156891 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:55.156780 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a20006f4-3771-49c5-9516-cb23ad68fcc3-service-ca" (OuterVolumeSpecName: "service-ca") pod "a20006f4-3771-49c5-9516-cb23ad68fcc3" (UID: "a20006f4-3771-49c5-9516-cb23ad68fcc3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 00:42:55.156891 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:55.156786 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a20006f4-3771-49c5-9516-cb23ad68fcc3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a20006f4-3771-49c5-9516-cb23ad68fcc3" (UID: "a20006f4-3771-49c5-9516-cb23ad68fcc3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 00:42:55.158976 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:55.158942 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a20006f4-3771-49c5-9516-cb23ad68fcc3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a20006f4-3771-49c5-9516-cb23ad68fcc3" (UID: "a20006f4-3771-49c5-9516-cb23ad68fcc3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 00:42:55.158976 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:55.158950 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a20006f4-3771-49c5-9516-cb23ad68fcc3-kube-api-access-gc655" (OuterVolumeSpecName: "kube-api-access-gc655") pod "a20006f4-3771-49c5-9516-cb23ad68fcc3" (UID: "a20006f4-3771-49c5-9516-cb23ad68fcc3"). InnerVolumeSpecName "kube-api-access-gc655". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 00:42:55.158976 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:55.158964 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a20006f4-3771-49c5-9516-cb23ad68fcc3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a20006f4-3771-49c5-9516-cb23ad68fcc3" (UID: "a20006f4-3771-49c5-9516-cb23ad68fcc3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 00:42:55.257509 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:55.257476 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc655\" (UniqueName: \"kubernetes.io/projected/a20006f4-3771-49c5-9516-cb23ad68fcc3-kube-api-access-gc655\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:42:55.257509 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:55.257504 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a20006f4-3771-49c5-9516-cb23ad68fcc3-console-config\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:42:55.257509 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:55.257514 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a20006f4-3771-49c5-9516-cb23ad68fcc3-console-serving-cert\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:42:55.257760 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:55.257522 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a20006f4-3771-49c5-9516-cb23ad68fcc3-service-ca\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:42:55.257760 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:55.257531 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a20006f4-3771-49c5-9516-cb23ad68fcc3-oauth-serving-cert\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:42:55.257760 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:55.257539 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a20006f4-3771-49c5-9516-cb23ad68fcc3-console-oauth-config\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:42:55.855210 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:55.855174 2580 generic.go:356] "Generic (PLEG): container finished" podID="a20006f4-3771-49c5-9516-cb23ad68fcc3" containerID="7a7ce70b026c3776c9482873c884c58b9f8b8428996343a5f0f86506d88f852d" exitCode=0 Jul 18 00:42:55.855643 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:55.855222 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f855dfbd7-nzqwb" Jul 18 00:42:55.855643 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:55.855234 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f855dfbd7-nzqwb" event={"ID":"a20006f4-3771-49c5-9516-cb23ad68fcc3","Type":"ContainerDied","Data":"7a7ce70b026c3776c9482873c884c58b9f8b8428996343a5f0f86506d88f852d"} Jul 18 00:42:55.855643 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:55.855274 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f855dfbd7-nzqwb" event={"ID":"a20006f4-3771-49c5-9516-cb23ad68fcc3","Type":"ContainerDied","Data":"c7ac5147fdec6bcae0f8878b016ac7cce503f977626e520a9bcafe631b509d3e"} Jul 18 00:42:55.855643 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:55.855293 2580 scope.go:117] "RemoveContainer" containerID="7a7ce70b026c3776c9482873c884c58b9f8b8428996343a5f0f86506d88f852d" Jul 18 00:42:55.864141 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:55.864123 2580 scope.go:117] "RemoveContainer" containerID="7a7ce70b026c3776c9482873c884c58b9f8b8428996343a5f0f86506d88f852d" Jul 18 00:42:55.864406 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:42:55.864387 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a7ce70b026c3776c9482873c884c58b9f8b8428996343a5f0f86506d88f852d\": container with ID starting with 7a7ce70b026c3776c9482873c884c58b9f8b8428996343a5f0f86506d88f852d not found: ID does not exist" containerID="7a7ce70b026c3776c9482873c884c58b9f8b8428996343a5f0f86506d88f852d" Jul 18 00:42:55.864444 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:55.864414 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a7ce70b026c3776c9482873c884c58b9f8b8428996343a5f0f86506d88f852d"} err="failed to get container status \"7a7ce70b026c3776c9482873c884c58b9f8b8428996343a5f0f86506d88f852d\": rpc error: code = NotFound desc = could not find container \"7a7ce70b026c3776c9482873c884c58b9f8b8428996343a5f0f86506d88f852d\": container with ID starting with 7a7ce70b026c3776c9482873c884c58b9f8b8428996343a5f0f86506d88f852d not found: ID does not exist" Jul 18 00:42:55.876985 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:55.876959 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f855dfbd7-nzqwb"] Jul 18 00:42:55.893544 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:55.891538 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f855dfbd7-nzqwb"] Jul 18 00:42:56.147885 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:42:56.147805 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a20006f4-3771-49c5-9516-cb23ad68fcc3" path="/var/lib/kubelet/pods/a20006f4-3771-49c5-9516-cb23ad68fcc3/volumes" Jul 18 00:43:36.334650 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.334569 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-65989b4df6-95szt"] Jul 18 00:43:36.335117 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.334955 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a20006f4-3771-49c5-9516-cb23ad68fcc3" containerName="console" Jul 18 00:43:36.335117 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.334974 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a20006f4-3771-49c5-9516-cb23ad68fcc3" containerName="console" Jul 18 00:43:36.335117 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.335042 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a20006f4-3771-49c5-9516-cb23ad68fcc3" containerName="console" Jul 18 00:43:36.338150 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.338130 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-65989b4df6-95szt" Jul 18 00:43:36.343727 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.342623 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-zgvcw\"" Jul 18 00:43:36.343727 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.342874 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 18 00:43:36.343727 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.343150 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 18 00:43:36.343727 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.343389 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 18 00:43:36.343727 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.343576 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 18 00:43:36.346988 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.346967 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 18 00:43:36.356283 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.356258 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-65989b4df6-95szt"] Jul 18 00:43:36.360708 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.360682 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 18 00:43:36.462489 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.462452 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a6f19b7d-7b8c-4e65-ae49-a284bd1191e1-federate-client-tls\") pod \"telemeter-client-65989b4df6-95szt\" (UID: \"a6f19b7d-7b8c-4e65-ae49-a284bd1191e1\") " pod="openshift-monitoring/telemeter-client-65989b4df6-95szt" Jul 18 00:43:36.462720 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.462503 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6f19b7d-7b8c-4e65-ae49-a284bd1191e1-serving-certs-ca-bundle\") pod \"telemeter-client-65989b4df6-95szt\" (UID: \"a6f19b7d-7b8c-4e65-ae49-a284bd1191e1\") " pod="openshift-monitoring/telemeter-client-65989b4df6-95szt" Jul 18 00:43:36.462720 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.462532 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a6f19b7d-7b8c-4e65-ae49-a284bd1191e1-telemeter-client-tls\") pod \"telemeter-client-65989b4df6-95szt\" (UID: \"a6f19b7d-7b8c-4e65-ae49-a284bd1191e1\") " pod="openshift-monitoring/telemeter-client-65989b4df6-95szt" Jul 18 00:43:36.462720 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.462554 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvkfv\" (UniqueName: \"kubernetes.io/projected/a6f19b7d-7b8c-4e65-ae49-a284bd1191e1-kube-api-access-tvkfv\") pod \"telemeter-client-65989b4df6-95szt\" (UID: \"a6f19b7d-7b8c-4e65-ae49-a284bd1191e1\") " pod="openshift-monitoring/telemeter-client-65989b4df6-95szt" Jul 18 00:43:36.462720 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.462589 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a6f19b7d-7b8c-4e65-ae49-a284bd1191e1-secret-telemeter-client\") pod \"telemeter-client-65989b4df6-95szt\" (UID: \"a6f19b7d-7b8c-4e65-ae49-a284bd1191e1\") " pod="openshift-monitoring/telemeter-client-65989b4df6-95szt" Jul 18 00:43:36.462720 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.462613 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6f19b7d-7b8c-4e65-ae49-a284bd1191e1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65989b4df6-95szt\" (UID: \"a6f19b7d-7b8c-4e65-ae49-a284bd1191e1\") " pod="openshift-monitoring/telemeter-client-65989b4df6-95szt" Jul 18 00:43:36.462962 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.462755 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6f19b7d-7b8c-4e65-ae49-a284bd1191e1-metrics-client-ca\") pod \"telemeter-client-65989b4df6-95szt\" (UID: \"a6f19b7d-7b8c-4e65-ae49-a284bd1191e1\") " pod="openshift-monitoring/telemeter-client-65989b4df6-95szt" Jul 18 00:43:36.462962 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.462802 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a6f19b7d-7b8c-4e65-ae49-a284bd1191e1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65989b4df6-95szt\" (UID: \"a6f19b7d-7b8c-4e65-ae49-a284bd1191e1\") " pod="openshift-monitoring/telemeter-client-65989b4df6-95szt" Jul 18 00:43:36.563815 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.563771 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6f19b7d-7b8c-4e65-ae49-a284bd1191e1-serving-certs-ca-bundle\") pod \"telemeter-client-65989b4df6-95szt\" (UID: \"a6f19b7d-7b8c-4e65-ae49-a284bd1191e1\") " pod="openshift-monitoring/telemeter-client-65989b4df6-95szt" Jul 18 00:43:36.564017 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.563824 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a6f19b7d-7b8c-4e65-ae49-a284bd1191e1-telemeter-client-tls\") pod \"telemeter-client-65989b4df6-95szt\" (UID: \"a6f19b7d-7b8c-4e65-ae49-a284bd1191e1\") " pod="openshift-monitoring/telemeter-client-65989b4df6-95szt" Jul 18 00:43:36.564017 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.563955 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvkfv\" (UniqueName: \"kubernetes.io/projected/a6f19b7d-7b8c-4e65-ae49-a284bd1191e1-kube-api-access-tvkfv\") pod \"telemeter-client-65989b4df6-95szt\" (UID: \"a6f19b7d-7b8c-4e65-ae49-a284bd1191e1\") " pod="openshift-monitoring/telemeter-client-65989b4df6-95szt" Jul 18 00:43:36.564017 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.564004 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a6f19b7d-7b8c-4e65-ae49-a284bd1191e1-secret-telemeter-client\") pod \"telemeter-client-65989b4df6-95szt\" (UID: \"a6f19b7d-7b8c-4e65-ae49-a284bd1191e1\") " pod="openshift-monitoring/telemeter-client-65989b4df6-95szt" Jul 18 00:43:36.564184 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.564022 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6f19b7d-7b8c-4e65-ae49-a284bd1191e1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65989b4df6-95szt\" (UID: \"a6f19b7d-7b8c-4e65-ae49-a284bd1191e1\") " pod="openshift-monitoring/telemeter-client-65989b4df6-95szt" Jul 18 00:43:36.564184 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.564055 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6f19b7d-7b8c-4e65-ae49-a284bd1191e1-metrics-client-ca\") pod \"telemeter-client-65989b4df6-95szt\" (UID: \"a6f19b7d-7b8c-4e65-ae49-a284bd1191e1\") " pod="openshift-monitoring/telemeter-client-65989b4df6-95szt" Jul 18 00:43:36.564184 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.564076 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a6f19b7d-7b8c-4e65-ae49-a284bd1191e1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65989b4df6-95szt\" (UID: \"a6f19b7d-7b8c-4e65-ae49-a284bd1191e1\") " pod="openshift-monitoring/telemeter-client-65989b4df6-95szt" Jul 18 00:43:36.564184 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.564130 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a6f19b7d-7b8c-4e65-ae49-a284bd1191e1-federate-client-tls\") pod \"telemeter-client-65989b4df6-95szt\" (UID: \"a6f19b7d-7b8c-4e65-ae49-a284bd1191e1\") " pod="openshift-monitoring/telemeter-client-65989b4df6-95szt" Jul 18 00:43:36.564834 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.564807 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6f19b7d-7b8c-4e65-ae49-a284bd1191e1-serving-certs-ca-bundle\") pod \"telemeter-client-65989b4df6-95szt\" (UID: \"a6f19b7d-7b8c-4e65-ae49-a284bd1191e1\") " pod="openshift-monitoring/telemeter-client-65989b4df6-95szt" Jul 18 00:43:36.565042 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.565016 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6f19b7d-7b8c-4e65-ae49-a284bd1191e1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65989b4df6-95szt\" (UID: \"a6f19b7d-7b8c-4e65-ae49-a284bd1191e1\") " pod="openshift-monitoring/telemeter-client-65989b4df6-95szt" Jul 18 00:43:36.565299 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.565271 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6f19b7d-7b8c-4e65-ae49-a284bd1191e1-metrics-client-ca\") pod \"telemeter-client-65989b4df6-95szt\" (UID: \"a6f19b7d-7b8c-4e65-ae49-a284bd1191e1\") " pod="openshift-monitoring/telemeter-client-65989b4df6-95szt" Jul 18 00:43:36.566863 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.566836 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a6f19b7d-7b8c-4e65-ae49-a284bd1191e1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65989b4df6-95szt\" (UID: \"a6f19b7d-7b8c-4e65-ae49-a284bd1191e1\") " pod="openshift-monitoring/telemeter-client-65989b4df6-95szt" Jul 18 00:43:36.567115 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.567096 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a6f19b7d-7b8c-4e65-ae49-a284bd1191e1-secret-telemeter-client\") pod \"telemeter-client-65989b4df6-95szt\" (UID: \"a6f19b7d-7b8c-4e65-ae49-a284bd1191e1\") " pod="openshift-monitoring/telemeter-client-65989b4df6-95szt" Jul 18 00:43:36.567277 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.567257 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a6f19b7d-7b8c-4e65-ae49-a284bd1191e1-federate-client-tls\") pod \"telemeter-client-65989b4df6-95szt\" (UID: \"a6f19b7d-7b8c-4e65-ae49-a284bd1191e1\") " pod="openshift-monitoring/telemeter-client-65989b4df6-95szt" Jul 18 00:43:36.567277 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.567270 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a6f19b7d-7b8c-4e65-ae49-a284bd1191e1-telemeter-client-tls\") pod \"telemeter-client-65989b4df6-95szt\" (UID: \"a6f19b7d-7b8c-4e65-ae49-a284bd1191e1\") " pod="openshift-monitoring/telemeter-client-65989b4df6-95szt" Jul 18 00:43:36.572349 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.572326 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvkfv\" (UniqueName: \"kubernetes.io/projected/a6f19b7d-7b8c-4e65-ae49-a284bd1191e1-kube-api-access-tvkfv\") pod \"telemeter-client-65989b4df6-95szt\" (UID: \"a6f19b7d-7b8c-4e65-ae49-a284bd1191e1\") " pod="openshift-monitoring/telemeter-client-65989b4df6-95szt" Jul 18 00:43:36.649185 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.649101 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-65989b4df6-95szt" Jul 18 00:43:36.802482 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.802455 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-65989b4df6-95szt"] Jul 18 00:43:36.804825 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:43:36.804800 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f19b7d_7b8c_4e65_ae49_a284bd1191e1.slice/crio-1653cf63f576452e7b2d09b5a5cf6896a2d44178d6cccc33bb61b5cbc50ad360 WatchSource:0}: Error finding container 1653cf63f576452e7b2d09b5a5cf6896a2d44178d6cccc33bb61b5cbc50ad360: Status 404 returned error can't find the container with id 1653cf63f576452e7b2d09b5a5cf6896a2d44178d6cccc33bb61b5cbc50ad360 Jul 18 00:43:36.989533 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:36.989494 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65989b4df6-95szt" event={"ID":"a6f19b7d-7b8c-4e65-ae49-a284bd1191e1","Type":"ContainerStarted","Data":"1653cf63f576452e7b2d09b5a5cf6896a2d44178d6cccc33bb61b5cbc50ad360"} Jul 18 00:43:38.998925 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:38.998882 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65989b4df6-95szt" event={"ID":"a6f19b7d-7b8c-4e65-ae49-a284bd1191e1","Type":"ContainerStarted","Data":"0a68ac38542f52931505695e2af6ce897b6a83be5e3a49ece3d26b2ed71e17aa"} Jul 18 00:43:40.004188 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:40.004160 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65989b4df6-95szt" event={"ID":"a6f19b7d-7b8c-4e65-ae49-a284bd1191e1","Type":"ContainerStarted","Data":"ffcc83ea1b841e5bbf9ba630c1ed1528d86d34c2e4bdd4ae0649a2b3fb58857e"} Jul 18 00:43:40.004465 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:40.004214 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65989b4df6-95szt" event={"ID":"a6f19b7d-7b8c-4e65-ae49-a284bd1191e1","Type":"ContainerStarted","Data":"aece7dd17592625f4c493e77c457406a4422eca409e257d0834fd9833d3b70d1"} Jul 18 00:43:40.025637 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:40.025579 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-65989b4df6-95szt" podStartSLOduration=1.441279921 podStartE2EDuration="4.025558854s" totalImagesPullingTime="2.584278933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:43:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:43:40.025399042 +0000 UTC m=+180.377249387" watchObservedRunningTime="2026-07-18 00:43:40.025558854 +0000 UTC m=+180.377409200" Jul 18 00:43:40.887373 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:40.887336 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-774d594979-l46bt"] Jul 18 00:43:40.891710 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:40.891689 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774d594979-l46bt" Jul 18 00:43:40.904909 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:40.904867 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-774d594979-l46bt"] Jul 18 00:43:41.003309 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:41.003269 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26076f2f-bff4-4473-ab02-d298435957fd-trusted-ca-bundle\") pod \"console-774d594979-l46bt\" (UID: \"26076f2f-bff4-4473-ab02-d298435957fd\") " pod="openshift-console/console-774d594979-l46bt" Jul 18 00:43:41.003489 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:41.003326 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26076f2f-bff4-4473-ab02-d298435957fd-service-ca\") pod \"console-774d594979-l46bt\" (UID: \"26076f2f-bff4-4473-ab02-d298435957fd\") " pod="openshift-console/console-774d594979-l46bt" Jul 18 00:43:41.003489 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:41.003351 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26076f2f-bff4-4473-ab02-d298435957fd-oauth-serving-cert\") pod \"console-774d594979-l46bt\" (UID: \"26076f2f-bff4-4473-ab02-d298435957fd\") " pod="openshift-console/console-774d594979-l46bt" Jul 18 00:43:41.003489 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:41.003374 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26076f2f-bff4-4473-ab02-d298435957fd-console-oauth-config\") pod \"console-774d594979-l46bt\" (UID: \"26076f2f-bff4-4473-ab02-d298435957fd\") " pod="openshift-console/console-774d594979-l46bt" Jul 18 00:43:41.003489 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:41.003399 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26076f2f-bff4-4473-ab02-d298435957fd-console-serving-cert\") pod \"console-774d594979-l46bt\" (UID: \"26076f2f-bff4-4473-ab02-d298435957fd\") " pod="openshift-console/console-774d594979-l46bt" Jul 18 00:43:41.003489 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:41.003421 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj54n\" (UniqueName: \"kubernetes.io/projected/26076f2f-bff4-4473-ab02-d298435957fd-kube-api-access-dj54n\") pod \"console-774d594979-l46bt\" (UID: \"26076f2f-bff4-4473-ab02-d298435957fd\") " pod="openshift-console/console-774d594979-l46bt" Jul 18 00:43:41.003489 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:41.003481 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26076f2f-bff4-4473-ab02-d298435957fd-console-config\") pod \"console-774d594979-l46bt\" (UID: \"26076f2f-bff4-4473-ab02-d298435957fd\") " pod="openshift-console/console-774d594979-l46bt" Jul 18 00:43:41.104287 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:41.104229 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26076f2f-bff4-4473-ab02-d298435957fd-trusted-ca-bundle\") pod \"console-774d594979-l46bt\" (UID: \"26076f2f-bff4-4473-ab02-d298435957fd\") " pod="openshift-console/console-774d594979-l46bt" Jul 18 00:43:41.104798 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:41.104432 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26076f2f-bff4-4473-ab02-d298435957fd-service-ca\") pod \"console-774d594979-l46bt\" (UID: \"26076f2f-bff4-4473-ab02-d298435957fd\") " pod="openshift-console/console-774d594979-l46bt" Jul 18 00:43:41.104798 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:41.104470 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26076f2f-bff4-4473-ab02-d298435957fd-oauth-serving-cert\") pod \"console-774d594979-l46bt\" (UID: \"26076f2f-bff4-4473-ab02-d298435957fd\") " pod="openshift-console/console-774d594979-l46bt" Jul 18 00:43:41.104798 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:41.104514 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26076f2f-bff4-4473-ab02-d298435957fd-console-oauth-config\") pod \"console-774d594979-l46bt\" (UID: \"26076f2f-bff4-4473-ab02-d298435957fd\") " pod="openshift-console/console-774d594979-l46bt" Jul 18 00:43:41.104798 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:41.104554 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26076f2f-bff4-4473-ab02-d298435957fd-console-serving-cert\") pod \"console-774d594979-l46bt\" (UID: \"26076f2f-bff4-4473-ab02-d298435957fd\") " pod="openshift-console/console-774d594979-l46bt" Jul 18 00:43:41.104798 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:41.104585 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj54n\" (UniqueName: \"kubernetes.io/projected/26076f2f-bff4-4473-ab02-d298435957fd-kube-api-access-dj54n\") pod \"console-774d594979-l46bt\" (UID: \"26076f2f-bff4-4473-ab02-d298435957fd\") " pod="openshift-console/console-774d594979-l46bt" Jul 18 00:43:41.104798 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:41.104650 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26076f2f-bff4-4473-ab02-d298435957fd-console-config\") pod \"console-774d594979-l46bt\" (UID: \"26076f2f-bff4-4473-ab02-d298435957fd\") " pod="openshift-console/console-774d594979-l46bt" Jul 18 00:43:41.105473 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:41.105444 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26076f2f-bff4-4473-ab02-d298435957fd-trusted-ca-bundle\") pod \"console-774d594979-l46bt\" (UID: \"26076f2f-bff4-4473-ab02-d298435957fd\") " pod="openshift-console/console-774d594979-l46bt" Jul 18 00:43:41.105582 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:41.105473 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26076f2f-bff4-4473-ab02-d298435957fd-console-config\") pod \"console-774d594979-l46bt\" (UID: \"26076f2f-bff4-4473-ab02-d298435957fd\") " pod="openshift-console/console-774d594979-l46bt" Jul 18 00:43:41.105582 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:41.105493 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26076f2f-bff4-4473-ab02-d298435957fd-oauth-serving-cert\") pod \"console-774d594979-l46bt\" (UID: \"26076f2f-bff4-4473-ab02-d298435957fd\") " pod="openshift-console/console-774d594979-l46bt" Jul 18 00:43:41.106006 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:41.105981 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26076f2f-bff4-4473-ab02-d298435957fd-service-ca\") pod \"console-774d594979-l46bt\" (UID: \"26076f2f-bff4-4473-ab02-d298435957fd\") " pod="openshift-console/console-774d594979-l46bt" Jul 18 00:43:41.107155 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:41.107134 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26076f2f-bff4-4473-ab02-d298435957fd-console-serving-cert\") pod \"console-774d594979-l46bt\" (UID: \"26076f2f-bff4-4473-ab02-d298435957fd\") " pod="openshift-console/console-774d594979-l46bt" Jul 18 00:43:41.107248 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:41.107186 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26076f2f-bff4-4473-ab02-d298435957fd-console-oauth-config\") pod \"console-774d594979-l46bt\" (UID: \"26076f2f-bff4-4473-ab02-d298435957fd\") " pod="openshift-console/console-774d594979-l46bt" Jul 18 00:43:41.113211 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:41.113186 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj54n\" (UniqueName: \"kubernetes.io/projected/26076f2f-bff4-4473-ab02-d298435957fd-kube-api-access-dj54n\") pod \"console-774d594979-l46bt\" (UID: \"26076f2f-bff4-4473-ab02-d298435957fd\") " pod="openshift-console/console-774d594979-l46bt" Jul 18 00:43:41.200215 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:41.200109 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774d594979-l46bt" Jul 18 00:43:41.332078 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:41.332054 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-774d594979-l46bt"] Jul 18 00:43:41.334360 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:43:41.334334 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26076f2f_bff4_4473_ab02_d298435957fd.slice/crio-d68bd5fd36b05fecb0d20b42325d388d4ed555c2dd81d2eca03903777daf3b37 WatchSource:0}: Error finding container d68bd5fd36b05fecb0d20b42325d388d4ed555c2dd81d2eca03903777daf3b37: Status 404 returned error can't find the container with id d68bd5fd36b05fecb0d20b42325d388d4ed555c2dd81d2eca03903777daf3b37 Jul 18 00:43:42.012523 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:42.012488 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774d594979-l46bt" event={"ID":"26076f2f-bff4-4473-ab02-d298435957fd","Type":"ContainerStarted","Data":"f5ccff244be9190d60e8eb4af45b0fd4d57bd7131bcd5079a7d78ea03ea3496a"} Jul 18 00:43:42.012723 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:42.012541 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774d594979-l46bt" event={"ID":"26076f2f-bff4-4473-ab02-d298435957fd","Type":"ContainerStarted","Data":"d68bd5fd36b05fecb0d20b42325d388d4ed555c2dd81d2eca03903777daf3b37"} Jul 18 00:43:42.030322 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:42.030272 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-774d594979-l46bt" podStartSLOduration=2.030253409 podStartE2EDuration="2.030253409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:43:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:43:42.028841186 +0000 UTC m=+182.380691525" watchObservedRunningTime="2026-07-18 00:43:42.030253409 +0000 UTC m=+182.382103755" Jul 18 00:43:51.200373 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:51.200329 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-774d594979-l46bt" Jul 18 00:43:51.200891 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:51.200400 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-774d594979-l46bt" Jul 18 00:43:51.205114 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:51.205095 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-774d594979-l46bt" Jul 18 00:43:52.049797 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:52.049766 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-774d594979-l46bt" Jul 18 00:43:52.093690 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:43:52.093639 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d595d7cbd-lxsw2"] Jul 18 00:44:17.114393 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:17.114321 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7d595d7cbd-lxsw2" podUID="3f188b37-4be7-4fd1-b4c9-65021bba45ca" containerName="console" containerID="cri-o://6d15db08956636848874f7568a93a36fdfe84b92aa66213c93fc2c5a0f5e0c7e" gracePeriod=15 Jul 18 00:44:17.361841 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:17.361818 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d595d7cbd-lxsw2" Jul 18 00:44:17.484998 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:17.484966 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f188b37-4be7-4fd1-b4c9-65021bba45ca-service-ca\") pod \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\" (UID: \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\") " Jul 18 00:44:17.485197 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:17.485008 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f188b37-4be7-4fd1-b4c9-65021bba45ca-console-serving-cert\") pod \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\" (UID: \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\") " Jul 18 00:44:17.485197 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:17.485029 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f188b37-4be7-4fd1-b4c9-65021bba45ca-oauth-serving-cert\") pod \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\" (UID: \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\") " Jul 18 00:44:17.485197 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:17.485064 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f188b37-4be7-4fd1-b4c9-65021bba45ca-console-config\") pod \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\" (UID: \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\") " Jul 18 00:44:17.485374 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:17.485216 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdnhd\" (UniqueName: \"kubernetes.io/projected/3f188b37-4be7-4fd1-b4c9-65021bba45ca-kube-api-access-jdnhd\") pod \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\" (UID: \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\") " Jul 18 00:44:17.485374 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:17.485262 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f188b37-4be7-4fd1-b4c9-65021bba45ca-trusted-ca-bundle\") pod \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\" (UID: \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\") " Jul 18 00:44:17.485374 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:17.485291 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f188b37-4be7-4fd1-b4c9-65021bba45ca-console-oauth-config\") pod \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\" (UID: \"3f188b37-4be7-4fd1-b4c9-65021bba45ca\") " Jul 18 00:44:17.485524 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:17.485420 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f188b37-4be7-4fd1-b4c9-65021bba45ca-service-ca" (OuterVolumeSpecName: "service-ca") pod "3f188b37-4be7-4fd1-b4c9-65021bba45ca" (UID: "3f188b37-4be7-4fd1-b4c9-65021bba45ca"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 00:44:17.485524 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:17.485504 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f188b37-4be7-4fd1-b4c9-65021bba45ca-console-config" (OuterVolumeSpecName: "console-config") pod "3f188b37-4be7-4fd1-b4c9-65021bba45ca" (UID: "3f188b37-4be7-4fd1-b4c9-65021bba45ca"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 00:44:17.485634 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:17.485522 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f188b37-4be7-4fd1-b4c9-65021bba45ca-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3f188b37-4be7-4fd1-b4c9-65021bba45ca" (UID: "3f188b37-4be7-4fd1-b4c9-65021bba45ca"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 00:44:17.485705 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:17.485659 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f188b37-4be7-4fd1-b4c9-65021bba45ca-service-ca\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:44:17.485825 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:17.485802 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f188b37-4be7-4fd1-b4c9-65021bba45ca-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3f188b37-4be7-4fd1-b4c9-65021bba45ca" (UID: "3f188b37-4be7-4fd1-b4c9-65021bba45ca"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 00:44:17.487402 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:17.487372 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f188b37-4be7-4fd1-b4c9-65021bba45ca-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3f188b37-4be7-4fd1-b4c9-65021bba45ca" (UID: "3f188b37-4be7-4fd1-b4c9-65021bba45ca"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 00:44:17.487512 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:17.487482 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f188b37-4be7-4fd1-b4c9-65021bba45ca-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3f188b37-4be7-4fd1-b4c9-65021bba45ca" (UID: "3f188b37-4be7-4fd1-b4c9-65021bba45ca"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 00:44:17.487553 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:17.487523 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f188b37-4be7-4fd1-b4c9-65021bba45ca-kube-api-access-jdnhd" (OuterVolumeSpecName: "kube-api-access-jdnhd") pod "3f188b37-4be7-4fd1-b4c9-65021bba45ca" (UID: "3f188b37-4be7-4fd1-b4c9-65021bba45ca"). InnerVolumeSpecName "kube-api-access-jdnhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 00:44:17.586961 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:17.586906 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f188b37-4be7-4fd1-b4c9-65021bba45ca-console-config\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:44:17.586961 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:17.586954 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdnhd\" (UniqueName: \"kubernetes.io/projected/3f188b37-4be7-4fd1-b4c9-65021bba45ca-kube-api-access-jdnhd\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:44:17.586961 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:17.586965 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f188b37-4be7-4fd1-b4c9-65021bba45ca-trusted-ca-bundle\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:44:17.586961 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:17.586974 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f188b37-4be7-4fd1-b4c9-65021bba45ca-console-oauth-config\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:44:17.587230 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:17.586982 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f188b37-4be7-4fd1-b4c9-65021bba45ca-console-serving-cert\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:44:17.587230 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:17.586991 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f188b37-4be7-4fd1-b4c9-65021bba45ca-oauth-serving-cert\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:44:18.133238 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:18.133208 2580 generic.go:356] "Generic (PLEG): container finished" podID="3f188b37-4be7-4fd1-b4c9-65021bba45ca" containerID="6d15db08956636848874f7568a93a36fdfe84b92aa66213c93fc2c5a0f5e0c7e" exitCode=0 Jul 18 00:44:18.133640 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:18.133247 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d595d7cbd-lxsw2" event={"ID":"3f188b37-4be7-4fd1-b4c9-65021bba45ca","Type":"ContainerDied","Data":"6d15db08956636848874f7568a93a36fdfe84b92aa66213c93fc2c5a0f5e0c7e"} Jul 18 00:44:18.133640 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:18.133281 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d595d7cbd-lxsw2" Jul 18 00:44:18.133640 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:18.133316 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d595d7cbd-lxsw2" event={"ID":"3f188b37-4be7-4fd1-b4c9-65021bba45ca","Type":"ContainerDied","Data":"6c0bca712c5e809bd5099603ebb9902854806a938d5738e730df195d9449f17a"} Jul 18 00:44:18.133640 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:18.133338 2580 scope.go:117] "RemoveContainer" containerID="6d15db08956636848874f7568a93a36fdfe84b92aa66213c93fc2c5a0f5e0c7e" Jul 18 00:44:18.141956 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:18.141940 2580 scope.go:117] "RemoveContainer" containerID="6d15db08956636848874f7568a93a36fdfe84b92aa66213c93fc2c5a0f5e0c7e" Jul 18 00:44:18.142216 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:44:18.142189 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d15db08956636848874f7568a93a36fdfe84b92aa66213c93fc2c5a0f5e0c7e\": container with ID starting with 6d15db08956636848874f7568a93a36fdfe84b92aa66213c93fc2c5a0f5e0c7e not found: ID does not exist" containerID="6d15db08956636848874f7568a93a36fdfe84b92aa66213c93fc2c5a0f5e0c7e" Jul 18 00:44:18.142262 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:18.142222 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d15db08956636848874f7568a93a36fdfe84b92aa66213c93fc2c5a0f5e0c7e"} err="failed to get container status \"6d15db08956636848874f7568a93a36fdfe84b92aa66213c93fc2c5a0f5e0c7e\": rpc error: code = NotFound desc = could not find container \"6d15db08956636848874f7568a93a36fdfe84b92aa66213c93fc2c5a0f5e0c7e\": container with ID starting with 6d15db08956636848874f7568a93a36fdfe84b92aa66213c93fc2c5a0f5e0c7e not found: ID does not exist" Jul 18 00:44:18.152894 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:18.152871 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d595d7cbd-lxsw2"] Jul 18 00:44:18.156867 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:18.156845 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d595d7cbd-lxsw2"] Jul 18 00:44:20.147839 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:20.147804 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f188b37-4be7-4fd1-b4c9-65021bba45ca" path="/var/lib/kubelet/pods/3f188b37-4be7-4fd1-b4c9-65021bba45ca/volumes" Jul 18 00:44:47.056913 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:47.056873 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65dd5fc6d-9rctc"] Jul 18 00:44:47.057281 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:47.057127 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f188b37-4be7-4fd1-b4c9-65021bba45ca" containerName="console" Jul 18 00:44:47.057281 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:47.057139 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f188b37-4be7-4fd1-b4c9-65021bba45ca" containerName="console" Jul 18 00:44:47.057281 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:47.057183 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f188b37-4be7-4fd1-b4c9-65021bba45ca" containerName="console" Jul 18 00:44:47.061054 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:47.061039 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65dd5fc6d-9rctc" Jul 18 00:44:47.072255 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:47.072227 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65dd5fc6d-9rctc"] Jul 18 00:44:47.219045 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:47.218995 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f87668bd-29be-47d3-a838-44177f22d563-console-config\") pod \"console-65dd5fc6d-9rctc\" (UID: \"f87668bd-29be-47d3-a838-44177f22d563\") " pod="openshift-console/console-65dd5fc6d-9rctc" Jul 18 00:44:47.219260 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:47.219056 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f87668bd-29be-47d3-a838-44177f22d563-trusted-ca-bundle\") pod \"console-65dd5fc6d-9rctc\" (UID: \"f87668bd-29be-47d3-a838-44177f22d563\") " pod="openshift-console/console-65dd5fc6d-9rctc" Jul 18 00:44:47.219260 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:47.219119 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f87668bd-29be-47d3-a838-44177f22d563-console-serving-cert\") pod \"console-65dd5fc6d-9rctc\" (UID: \"f87668bd-29be-47d3-a838-44177f22d563\") " pod="openshift-console/console-65dd5fc6d-9rctc" Jul 18 00:44:47.219260 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:47.219161 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f87668bd-29be-47d3-a838-44177f22d563-oauth-serving-cert\") pod \"console-65dd5fc6d-9rctc\" (UID: \"f87668bd-29be-47d3-a838-44177f22d563\") " pod="openshift-console/console-65dd5fc6d-9rctc" Jul 18 00:44:47.219260 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:47.219186 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f87668bd-29be-47d3-a838-44177f22d563-console-oauth-config\") pod \"console-65dd5fc6d-9rctc\" (UID: \"f87668bd-29be-47d3-a838-44177f22d563\") " pod="openshift-console/console-65dd5fc6d-9rctc" Jul 18 00:44:47.219260 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:47.219219 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snkmz\" (UniqueName: \"kubernetes.io/projected/f87668bd-29be-47d3-a838-44177f22d563-kube-api-access-snkmz\") pod \"console-65dd5fc6d-9rctc\" (UID: \"f87668bd-29be-47d3-a838-44177f22d563\") " pod="openshift-console/console-65dd5fc6d-9rctc" Jul 18 00:44:47.219426 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:47.219258 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f87668bd-29be-47d3-a838-44177f22d563-service-ca\") pod \"console-65dd5fc6d-9rctc\" (UID: \"f87668bd-29be-47d3-a838-44177f22d563\") " pod="openshift-console/console-65dd5fc6d-9rctc" Jul 18 00:44:47.320583 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:47.320486 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snkmz\" (UniqueName: \"kubernetes.io/projected/f87668bd-29be-47d3-a838-44177f22d563-kube-api-access-snkmz\") pod \"console-65dd5fc6d-9rctc\" (UID: \"f87668bd-29be-47d3-a838-44177f22d563\") " pod="openshift-console/console-65dd5fc6d-9rctc" Jul 18 00:44:47.320583 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:47.320529 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f87668bd-29be-47d3-a838-44177f22d563-service-ca\") pod \"console-65dd5fc6d-9rctc\" (UID: \"f87668bd-29be-47d3-a838-44177f22d563\") " pod="openshift-console/console-65dd5fc6d-9rctc" Jul 18 00:44:47.320583 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:47.320560 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f87668bd-29be-47d3-a838-44177f22d563-console-config\") pod \"console-65dd5fc6d-9rctc\" (UID: \"f87668bd-29be-47d3-a838-44177f22d563\") " pod="openshift-console/console-65dd5fc6d-9rctc" Jul 18 00:44:47.320583 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:47.320589 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f87668bd-29be-47d3-a838-44177f22d563-trusted-ca-bundle\") pod \"console-65dd5fc6d-9rctc\" (UID: \"f87668bd-29be-47d3-a838-44177f22d563\") " pod="openshift-console/console-65dd5fc6d-9rctc" Jul 18 00:44:47.320941 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:47.320617 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f87668bd-29be-47d3-a838-44177f22d563-console-serving-cert\") pod \"console-65dd5fc6d-9rctc\" (UID: \"f87668bd-29be-47d3-a838-44177f22d563\") " pod="openshift-console/console-65dd5fc6d-9rctc" Jul 18 00:44:47.320941 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:47.320642 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f87668bd-29be-47d3-a838-44177f22d563-oauth-serving-cert\") pod \"console-65dd5fc6d-9rctc\" (UID: \"f87668bd-29be-47d3-a838-44177f22d563\") " pod="openshift-console/console-65dd5fc6d-9rctc" Jul 18 00:44:47.320941 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:47.320661 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f87668bd-29be-47d3-a838-44177f22d563-console-oauth-config\") pod \"console-65dd5fc6d-9rctc\" (UID: \"f87668bd-29be-47d3-a838-44177f22d563\") " pod="openshift-console/console-65dd5fc6d-9rctc" Jul 18 00:44:47.321397 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:47.321322 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f87668bd-29be-47d3-a838-44177f22d563-service-ca\") pod \"console-65dd5fc6d-9rctc\" (UID: \"f87668bd-29be-47d3-a838-44177f22d563\") " pod="openshift-console/console-65dd5fc6d-9rctc" Jul 18 00:44:47.321397 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:47.321423 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f87668bd-29be-47d3-a838-44177f22d563-console-config\") pod \"console-65dd5fc6d-9rctc\" (UID: \"f87668bd-29be-47d3-a838-44177f22d563\") " pod="openshift-console/console-65dd5fc6d-9rctc" Jul 18 00:44:47.321637 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:47.321519 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f87668bd-29be-47d3-a838-44177f22d563-trusted-ca-bundle\") pod \"console-65dd5fc6d-9rctc\" (UID: \"f87668bd-29be-47d3-a838-44177f22d563\") " pod="openshift-console/console-65dd5fc6d-9rctc" Jul 18 00:44:47.321730 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:47.321644 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f87668bd-29be-47d3-a838-44177f22d563-oauth-serving-cert\") pod \"console-65dd5fc6d-9rctc\" (UID: \"f87668bd-29be-47d3-a838-44177f22d563\") " pod="openshift-console/console-65dd5fc6d-9rctc" Jul 18 00:44:47.323261 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:47.323234 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f87668bd-29be-47d3-a838-44177f22d563-console-oauth-config\") pod \"console-65dd5fc6d-9rctc\" (UID: \"f87668bd-29be-47d3-a838-44177f22d563\") " pod="openshift-console/console-65dd5fc6d-9rctc" Jul 18 00:44:47.323367 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:47.323326 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f87668bd-29be-47d3-a838-44177f22d563-console-serving-cert\") pod \"console-65dd5fc6d-9rctc\" (UID: \"f87668bd-29be-47d3-a838-44177f22d563\") " pod="openshift-console/console-65dd5fc6d-9rctc" Jul 18 00:44:47.328704 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:47.328663 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snkmz\" (UniqueName: \"kubernetes.io/projected/f87668bd-29be-47d3-a838-44177f22d563-kube-api-access-snkmz\") pod \"console-65dd5fc6d-9rctc\" (UID: \"f87668bd-29be-47d3-a838-44177f22d563\") " pod="openshift-console/console-65dd5fc6d-9rctc" Jul 18 00:44:47.369658 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:47.369620 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65dd5fc6d-9rctc" Jul 18 00:44:47.499853 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:47.499770 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65dd5fc6d-9rctc"] Jul 18 00:44:47.502451 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:44:47.502426 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf87668bd_29be_47d3_a838_44177f22d563.slice/crio-d789947aba873d67ef0ee9a5fb5c7be19d59303423951a42cbe522b5120b003b WatchSource:0}: Error finding container d789947aba873d67ef0ee9a5fb5c7be19d59303423951a42cbe522b5120b003b: Status 404 returned error can't find the container with id d789947aba873d67ef0ee9a5fb5c7be19d59303423951a42cbe522b5120b003b Jul 18 00:44:48.236158 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:48.236118 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65dd5fc6d-9rctc" event={"ID":"f87668bd-29be-47d3-a838-44177f22d563","Type":"ContainerStarted","Data":"a3517d50e1ec6551fc01d904261694721b1c299358b3a2359c4fd53d4d39ae34"} Jul 18 00:44:48.236692 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:48.236180 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65dd5fc6d-9rctc" event={"ID":"f87668bd-29be-47d3-a838-44177f22d563","Type":"ContainerStarted","Data":"d789947aba873d67ef0ee9a5fb5c7be19d59303423951a42cbe522b5120b003b"} Jul 18 00:44:48.253034 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:48.252966 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65dd5fc6d-9rctc" podStartSLOduration=1.252949904 podStartE2EDuration="1.252949904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:44:48.252219708 +0000 UTC m=+248.604070055" watchObservedRunningTime="2026-07-18 00:44:48.252949904 +0000 UTC m=+248.604800249" Jul 18 00:44:57.369847 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:57.369804 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65dd5fc6d-9rctc" Jul 18 00:44:57.370231 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:57.369884 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65dd5fc6d-9rctc" Jul 18 00:44:57.374452 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:57.374432 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65dd5fc6d-9rctc" Jul 18 00:44:58.272619 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:58.272589 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65dd5fc6d-9rctc" Jul 18 00:44:58.321562 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:44:58.321527 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-774d594979-l46bt"] Jul 18 00:45:23.342537 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:23.342494 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-774d594979-l46bt" podUID="26076f2f-bff4-4473-ab02-d298435957fd" containerName="console" containerID="cri-o://f5ccff244be9190d60e8eb4af45b0fd4d57bd7131bcd5079a7d78ea03ea3496a" gracePeriod=15 Jul 18 00:45:23.586003 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:23.585976 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774d594979-l46bt" Jul 18 00:45:23.677778 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:23.677690 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26076f2f-bff4-4473-ab02-d298435957fd-console-serving-cert\") pod \"26076f2f-bff4-4473-ab02-d298435957fd\" (UID: \"26076f2f-bff4-4473-ab02-d298435957fd\") " Jul 18 00:45:23.677778 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:23.677737 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26076f2f-bff4-4473-ab02-d298435957fd-oauth-serving-cert\") pod \"26076f2f-bff4-4473-ab02-d298435957fd\" (UID: \"26076f2f-bff4-4473-ab02-d298435957fd\") " Jul 18 00:45:23.677778 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:23.677775 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dj54n\" (UniqueName: \"kubernetes.io/projected/26076f2f-bff4-4473-ab02-d298435957fd-kube-api-access-dj54n\") pod \"26076f2f-bff4-4473-ab02-d298435957fd\" (UID: \"26076f2f-bff4-4473-ab02-d298435957fd\") " Jul 18 00:45:23.678036 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:23.677808 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26076f2f-bff4-4473-ab02-d298435957fd-console-oauth-config\") pod \"26076f2f-bff4-4473-ab02-d298435957fd\" (UID: \"26076f2f-bff4-4473-ab02-d298435957fd\") " Jul 18 00:45:23.678036 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:23.677825 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26076f2f-bff4-4473-ab02-d298435957fd-service-ca\") pod \"26076f2f-bff4-4473-ab02-d298435957fd\" (UID: \"26076f2f-bff4-4473-ab02-d298435957fd\") " Jul 18 00:45:23.678036 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:23.677840 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26076f2f-bff4-4473-ab02-d298435957fd-console-config\") pod \"26076f2f-bff4-4473-ab02-d298435957fd\" (UID: \"26076f2f-bff4-4473-ab02-d298435957fd\") " Jul 18 00:45:23.678036 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:23.677856 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26076f2f-bff4-4473-ab02-d298435957fd-trusted-ca-bundle\") pod \"26076f2f-bff4-4473-ab02-d298435957fd\" (UID: \"26076f2f-bff4-4473-ab02-d298435957fd\") " Jul 18 00:45:23.678358 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:23.678299 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26076f2f-bff4-4473-ab02-d298435957fd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "26076f2f-bff4-4473-ab02-d298435957fd" (UID: "26076f2f-bff4-4473-ab02-d298435957fd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 00:45:23.678358 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:23.678317 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26076f2f-bff4-4473-ab02-d298435957fd-service-ca" (OuterVolumeSpecName: "service-ca") pod "26076f2f-bff4-4473-ab02-d298435957fd" (UID: "26076f2f-bff4-4473-ab02-d298435957fd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 00:45:23.678358 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:23.678305 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26076f2f-bff4-4473-ab02-d298435957fd-console-config" (OuterVolumeSpecName: "console-config") pod "26076f2f-bff4-4473-ab02-d298435957fd" (UID: "26076f2f-bff4-4473-ab02-d298435957fd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 00:45:23.678358 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:23.678354 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26076f2f-bff4-4473-ab02-d298435957fd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "26076f2f-bff4-4473-ab02-d298435957fd" (UID: "26076f2f-bff4-4473-ab02-d298435957fd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 00:45:23.680215 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:23.680176 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26076f2f-bff4-4473-ab02-d298435957fd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "26076f2f-bff4-4473-ab02-d298435957fd" (UID: "26076f2f-bff4-4473-ab02-d298435957fd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 00:45:23.680511 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:23.680489 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26076f2f-bff4-4473-ab02-d298435957fd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "26076f2f-bff4-4473-ab02-d298435957fd" (UID: "26076f2f-bff4-4473-ab02-d298435957fd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 00:45:23.680511 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:23.680498 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26076f2f-bff4-4473-ab02-d298435957fd-kube-api-access-dj54n" (OuterVolumeSpecName: "kube-api-access-dj54n") pod "26076f2f-bff4-4473-ab02-d298435957fd" (UID: "26076f2f-bff4-4473-ab02-d298435957fd"). InnerVolumeSpecName "kube-api-access-dj54n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 00:45:23.779006 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:23.778960 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj54n\" (UniqueName: \"kubernetes.io/projected/26076f2f-bff4-4473-ab02-d298435957fd-kube-api-access-dj54n\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:45:23.779006 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:23.778992 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26076f2f-bff4-4473-ab02-d298435957fd-console-oauth-config\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:45:23.779006 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:23.779002 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26076f2f-bff4-4473-ab02-d298435957fd-service-ca\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:45:23.779006 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:23.779010 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26076f2f-bff4-4473-ab02-d298435957fd-console-config\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:45:23.779006 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:23.779018 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26076f2f-bff4-4473-ab02-d298435957fd-trusted-ca-bundle\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:45:23.779291 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:23.779026 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26076f2f-bff4-4473-ab02-d298435957fd-console-serving-cert\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:45:23.779291 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:23.779034 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26076f2f-bff4-4473-ab02-d298435957fd-oauth-serving-cert\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:45:24.358692 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:24.358635 2580 generic.go:356] "Generic (PLEG): container finished" podID="26076f2f-bff4-4473-ab02-d298435957fd" containerID="f5ccff244be9190d60e8eb4af45b0fd4d57bd7131bcd5079a7d78ea03ea3496a" exitCode=0 Jul 18 00:45:24.359092 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:24.358702 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774d594979-l46bt" Jul 18 00:45:24.359092 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:24.358718 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774d594979-l46bt" event={"ID":"26076f2f-bff4-4473-ab02-d298435957fd","Type":"ContainerDied","Data":"f5ccff244be9190d60e8eb4af45b0fd4d57bd7131bcd5079a7d78ea03ea3496a"} Jul 18 00:45:24.359092 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:24.358762 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774d594979-l46bt" event={"ID":"26076f2f-bff4-4473-ab02-d298435957fd","Type":"ContainerDied","Data":"d68bd5fd36b05fecb0d20b42325d388d4ed555c2dd81d2eca03903777daf3b37"} Jul 18 00:45:24.359092 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:24.358787 2580 scope.go:117] "RemoveContainer" containerID="f5ccff244be9190d60e8eb4af45b0fd4d57bd7131bcd5079a7d78ea03ea3496a" Jul 18 00:45:24.366595 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:24.366576 2580 scope.go:117] "RemoveContainer" containerID="f5ccff244be9190d60e8eb4af45b0fd4d57bd7131bcd5079a7d78ea03ea3496a" Jul 18 00:45:24.366910 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:45:24.366892 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5ccff244be9190d60e8eb4af45b0fd4d57bd7131bcd5079a7d78ea03ea3496a\": container with ID starting with f5ccff244be9190d60e8eb4af45b0fd4d57bd7131bcd5079a7d78ea03ea3496a not found: ID does not exist" containerID="f5ccff244be9190d60e8eb4af45b0fd4d57bd7131bcd5079a7d78ea03ea3496a" Jul 18 00:45:24.366976 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:24.366925 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5ccff244be9190d60e8eb4af45b0fd4d57bd7131bcd5079a7d78ea03ea3496a"} err="failed to get container status \"f5ccff244be9190d60e8eb4af45b0fd4d57bd7131bcd5079a7d78ea03ea3496a\": rpc error: code = NotFound desc = could not find container \"f5ccff244be9190d60e8eb4af45b0fd4d57bd7131bcd5079a7d78ea03ea3496a\": container with ID starting with f5ccff244be9190d60e8eb4af45b0fd4d57bd7131bcd5079a7d78ea03ea3496a not found: ID does not exist" Jul 18 00:45:24.374093 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:24.374066 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-774d594979-l46bt"] Jul 18 00:45:24.379456 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:24.379432 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-774d594979-l46bt"] Jul 18 00:45:26.147538 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:26.147497 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26076f2f-bff4-4473-ab02-d298435957fd" path="/var/lib/kubelet/pods/26076f2f-bff4-4473-ab02-d298435957fd/volumes" Jul 18 00:45:34.658625 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:34.658589 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qhv6"] Jul 18 00:45:34.659030 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:34.658864 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26076f2f-bff4-4473-ab02-d298435957fd" containerName="console" Jul 18 00:45:34.659030 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:34.658876 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="26076f2f-bff4-4473-ab02-d298435957fd" containerName="console" Jul 18 00:45:34.659030 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:34.658929 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="26076f2f-bff4-4473-ab02-d298435957fd" containerName="console" Jul 18 00:45:34.661492 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:34.661476 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qhv6" Jul 18 00:45:34.663924 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:34.663901 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 18 00:45:34.663924 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:34.663922 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9rsk7\"" Jul 18 00:45:34.664903 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:34.664890 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 18 00:45:34.670352 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:34.670333 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qhv6"] Jul 18 00:45:34.759967 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:34.759913 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2f21341-8f01-4b4a-a6fa-4404072b4266-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qhv6\" (UID: \"a2f21341-8f01-4b4a-a6fa-4404072b4266\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qhv6" Jul 18 00:45:34.759967 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:34.759969 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2f21341-8f01-4b4a-a6fa-4404072b4266-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qhv6\" (UID: \"a2f21341-8f01-4b4a-a6fa-4404072b4266\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qhv6" Jul 18 00:45:34.760184 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:34.759998 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lwc4\" (UniqueName: \"kubernetes.io/projected/a2f21341-8f01-4b4a-a6fa-4404072b4266-kube-api-access-5lwc4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qhv6\" (UID: \"a2f21341-8f01-4b4a-a6fa-4404072b4266\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qhv6" Jul 18 00:45:34.860802 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:34.860751 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2f21341-8f01-4b4a-a6fa-4404072b4266-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qhv6\" (UID: \"a2f21341-8f01-4b4a-a6fa-4404072b4266\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qhv6" Jul 18 00:45:34.860802 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:34.860807 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2f21341-8f01-4b4a-a6fa-4404072b4266-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qhv6\" (UID: \"a2f21341-8f01-4b4a-a6fa-4404072b4266\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qhv6" Jul 18 00:45:34.861050 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:34.860849 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lwc4\" (UniqueName: \"kubernetes.io/projected/a2f21341-8f01-4b4a-a6fa-4404072b4266-kube-api-access-5lwc4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qhv6\" (UID: \"a2f21341-8f01-4b4a-a6fa-4404072b4266\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qhv6" Jul 18 00:45:34.861157 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:34.861137 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2f21341-8f01-4b4a-a6fa-4404072b4266-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qhv6\" (UID: \"a2f21341-8f01-4b4a-a6fa-4404072b4266\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qhv6" Jul 18 00:45:34.861216 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:34.861161 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2f21341-8f01-4b4a-a6fa-4404072b4266-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qhv6\" (UID: \"a2f21341-8f01-4b4a-a6fa-4404072b4266\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qhv6" Jul 18 00:45:34.869425 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:34.869394 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lwc4\" (UniqueName: \"kubernetes.io/projected/a2f21341-8f01-4b4a-a6fa-4404072b4266-kube-api-access-5lwc4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qhv6\" (UID: \"a2f21341-8f01-4b4a-a6fa-4404072b4266\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qhv6" Jul 18 00:45:34.970597 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:34.970561 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qhv6" Jul 18 00:45:35.099232 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:35.099087 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qhv6"] Jul 18 00:45:35.101583 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:45:35.101556 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f21341_8f01_4b4a_a6fa_4404072b4266.slice/crio-bd8ad9c959f3494d5368acb61d0dce8a01bcd12b6d6f36f461c659a48cf1e86d WatchSource:0}: Error finding container bd8ad9c959f3494d5368acb61d0dce8a01bcd12b6d6f36f461c659a48cf1e86d: Status 404 returned error can't find the container with id bd8ad9c959f3494d5368acb61d0dce8a01bcd12b6d6f36f461c659a48cf1e86d Jul 18 00:45:35.399571 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:35.399484 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qhv6" event={"ID":"a2f21341-8f01-4b4a-a6fa-4404072b4266","Type":"ContainerStarted","Data":"bd8ad9c959f3494d5368acb61d0dce8a01bcd12b6d6f36f461c659a48cf1e86d"} Jul 18 00:45:41.623101 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:41.623072 2580 kubelet.go:1648] "Image garbage collection succeeded" Jul 18 00:45:42.429081 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:42.429049 2580 generic.go:356] "Generic (PLEG): container finished" podID="a2f21341-8f01-4b4a-a6fa-4404072b4266" containerID="59f180c787228421264d3927a502cef577882f20aaca4f5beb514d3c7545f735" exitCode=0 Jul 18 00:45:42.429267 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:42.429102 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qhv6" event={"ID":"a2f21341-8f01-4b4a-a6fa-4404072b4266","Type":"ContainerDied","Data":"59f180c787228421264d3927a502cef577882f20aaca4f5beb514d3c7545f735"} Jul 18 00:45:42.430101 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:42.430084 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 00:45:45.442350 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:45.442315 2580 generic.go:356] "Generic (PLEG): container finished" podID="a2f21341-8f01-4b4a-a6fa-4404072b4266" containerID="d7381f3891d7112a549f2010426cc43f2f1de34fd95806123cad798115e4a163" exitCode=0 Jul 18 00:45:45.442744 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:45.442394 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qhv6" event={"ID":"a2f21341-8f01-4b4a-a6fa-4404072b4266","Type":"ContainerDied","Data":"d7381f3891d7112a549f2010426cc43f2f1de34fd95806123cad798115e4a163"} Jul 18 00:45:52.478446 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:52.478411 2580 generic.go:356] "Generic (PLEG): container finished" podID="a2f21341-8f01-4b4a-a6fa-4404072b4266" containerID="9d36e4d3f5f413deda75559a69c7d325654faebdbecf5ae7c8f5d2b1922a6dc4" exitCode=0 Jul 18 00:45:52.478945 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:52.478467 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qhv6" event={"ID":"a2f21341-8f01-4b4a-a6fa-4404072b4266","Type":"ContainerDied","Data":"9d36e4d3f5f413deda75559a69c7d325654faebdbecf5ae7c8f5d2b1922a6dc4"} Jul 18 00:45:53.607546 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:53.607517 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qhv6" Jul 18 00:45:53.719699 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:53.719632 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2f21341-8f01-4b4a-a6fa-4404072b4266-util\") pod \"a2f21341-8f01-4b4a-a6fa-4404072b4266\" (UID: \"a2f21341-8f01-4b4a-a6fa-4404072b4266\") " Jul 18 00:45:53.719869 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:53.719722 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2f21341-8f01-4b4a-a6fa-4404072b4266-bundle\") pod \"a2f21341-8f01-4b4a-a6fa-4404072b4266\" (UID: \"a2f21341-8f01-4b4a-a6fa-4404072b4266\") " Jul 18 00:45:53.719869 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:53.719798 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lwc4\" (UniqueName: \"kubernetes.io/projected/a2f21341-8f01-4b4a-a6fa-4404072b4266-kube-api-access-5lwc4\") pod \"a2f21341-8f01-4b4a-a6fa-4404072b4266\" (UID: \"a2f21341-8f01-4b4a-a6fa-4404072b4266\") " Jul 18 00:45:53.720448 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:53.720423 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f21341-8f01-4b4a-a6fa-4404072b4266-bundle" (OuterVolumeSpecName: "bundle") pod "a2f21341-8f01-4b4a-a6fa-4404072b4266" (UID: "a2f21341-8f01-4b4a-a6fa-4404072b4266"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 00:45:53.722131 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:53.722110 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f21341-8f01-4b4a-a6fa-4404072b4266-kube-api-access-5lwc4" (OuterVolumeSpecName: "kube-api-access-5lwc4") pod "a2f21341-8f01-4b4a-a6fa-4404072b4266" (UID: "a2f21341-8f01-4b4a-a6fa-4404072b4266"). InnerVolumeSpecName "kube-api-access-5lwc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 00:45:53.723916 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:53.723885 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f21341-8f01-4b4a-a6fa-4404072b4266-util" (OuterVolumeSpecName: "util") pod "a2f21341-8f01-4b4a-a6fa-4404072b4266" (UID: "a2f21341-8f01-4b4a-a6fa-4404072b4266"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 00:45:53.821327 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:53.821220 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2f21341-8f01-4b4a-a6fa-4404072b4266-bundle\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:45:53.821327 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:53.821268 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lwc4\" (UniqueName: \"kubernetes.io/projected/a2f21341-8f01-4b4a-a6fa-4404072b4266-kube-api-access-5lwc4\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:45:53.821327 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:53.821279 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2f21341-8f01-4b4a-a6fa-4404072b4266-util\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:45:54.486832 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:54.486802 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qhv6" event={"ID":"a2f21341-8f01-4b4a-a6fa-4404072b4266","Type":"ContainerDied","Data":"bd8ad9c959f3494d5368acb61d0dce8a01bcd12b6d6f36f461c659a48cf1e86d"} Jul 18 00:45:54.487009 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:54.486846 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd8ad9c959f3494d5368acb61d0dce8a01bcd12b6d6f36f461c659a48cf1e86d" Jul 18 00:45:54.487009 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:54.486860 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402qhv6" Jul 18 00:45:57.790688 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:57.790642 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9f42m"] Jul 18 00:45:57.791075 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:57.790913 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2f21341-8f01-4b4a-a6fa-4404072b4266" containerName="util" Jul 18 00:45:57.791075 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:57.790924 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2f21341-8f01-4b4a-a6fa-4404072b4266" containerName="util" Jul 18 00:45:57.791075 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:57.790931 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2f21341-8f01-4b4a-a6fa-4404072b4266" containerName="extract" Jul 18 00:45:57.791075 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:57.790937 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2f21341-8f01-4b4a-a6fa-4404072b4266" containerName="extract" Jul 18 00:45:57.791075 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:57.790948 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2f21341-8f01-4b4a-a6fa-4404072b4266" containerName="pull" Jul 18 00:45:57.791075 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:57.790953 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2f21341-8f01-4b4a-a6fa-4404072b4266" containerName="pull" Jul 18 00:45:57.791075 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:57.790998 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2f21341-8f01-4b4a-a6fa-4404072b4266" containerName="extract" Jul 18 00:45:57.826718 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:57.826661 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9f42m"] Jul 18 00:45:57.826871 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:57.826791 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9f42m" Jul 18 00:45:57.829530 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:57.829509 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-slgsh\"" Jul 18 00:45:57.829660 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:57.829539 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 18 00:45:57.829660 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:57.829592 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 18 00:45:57.953754 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:57.953718 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3cd10236-d5a0-4536-8330-0d35a0a76154-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-9f42m\" (UID: \"3cd10236-d5a0-4536-8330-0d35a0a76154\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9f42m" Jul 18 00:45:57.953898 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:57.953764 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxbrt\" (UniqueName: \"kubernetes.io/projected/3cd10236-d5a0-4536-8330-0d35a0a76154-kube-api-access-qxbrt\") pod \"cert-manager-operator-controller-manager-55c444bf86-9f42m\" (UID: \"3cd10236-d5a0-4536-8330-0d35a0a76154\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9f42m" Jul 18 00:45:58.054794 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:58.054705 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxbrt\" (UniqueName: \"kubernetes.io/projected/3cd10236-d5a0-4536-8330-0d35a0a76154-kube-api-access-qxbrt\") pod \"cert-manager-operator-controller-manager-55c444bf86-9f42m\" (UID: \"3cd10236-d5a0-4536-8330-0d35a0a76154\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9f42m" Jul 18 00:45:58.054794 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:58.054796 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3cd10236-d5a0-4536-8330-0d35a0a76154-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-9f42m\" (UID: \"3cd10236-d5a0-4536-8330-0d35a0a76154\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9f42m" Jul 18 00:45:58.055121 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:58.055106 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3cd10236-d5a0-4536-8330-0d35a0a76154-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-9f42m\" (UID: \"3cd10236-d5a0-4536-8330-0d35a0a76154\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9f42m" Jul 18 00:45:58.063574 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:58.063553 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxbrt\" (UniqueName: \"kubernetes.io/projected/3cd10236-d5a0-4536-8330-0d35a0a76154-kube-api-access-qxbrt\") pod \"cert-manager-operator-controller-manager-55c444bf86-9f42m\" (UID: \"3cd10236-d5a0-4536-8330-0d35a0a76154\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9f42m" Jul 18 00:45:58.135226 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:58.135199 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9f42m" Jul 18 00:45:58.262799 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:58.262725 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9f42m"] Jul 18 00:45:58.265794 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:45:58.265765 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd10236_d5a0_4536_8330_0d35a0a76154.slice/crio-4f1da04b003fbdcef9de027435f54c38843f5759b16a0488c19e11ae587d183b WatchSource:0}: Error finding container 4f1da04b003fbdcef9de027435f54c38843f5759b16a0488c19e11ae587d183b: Status 404 returned error can't find the container with id 4f1da04b003fbdcef9de027435f54c38843f5759b16a0488c19e11ae587d183b Jul 18 00:45:58.502655 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:45:58.502625 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9f42m" event={"ID":"3cd10236-d5a0-4536-8330-0d35a0a76154","Type":"ContainerStarted","Data":"4f1da04b003fbdcef9de027435f54c38843f5759b16a0488c19e11ae587d183b"} Jul 18 00:46:00.512138 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:00.512101 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9f42m" event={"ID":"3cd10236-d5a0-4536-8330-0d35a0a76154","Type":"ContainerStarted","Data":"6e086a50a0a590917409258e1eb86c582239d956dea2795cd0ac4ad1f5594e8b"} Jul 18 00:46:00.536716 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:00.536640 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9f42m" podStartSLOduration=1.9678145649999998 podStartE2EDuration="3.536621848s" totalImagesPullingTime="1.568807283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:45:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:46:00.534122641 +0000 UTC m=+320.885972986" watchObservedRunningTime="2026-07-18 00:46:00.536621848 +0000 UTC m=+320.888472192" Jul 18 00:46:01.971735 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:01.971703 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f477w5"] Jul 18 00:46:01.974007 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:01.973989 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f477w5" Jul 18 00:46:01.976798 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:01.976778 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 18 00:46:01.976911 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:01.976784 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 18 00:46:01.977632 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:01.977617 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9rsk7\"" Jul 18 00:46:01.987063 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:01.987038 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f477w5"] Jul 18 00:46:02.074263 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:02.074228 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qckh7"] Jul 18 00:46:02.076412 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:02.076392 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-qckh7" Jul 18 00:46:02.078840 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:02.078811 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-qbd9c\"" Jul 18 00:46:02.078974 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:02.078931 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 18 00:46:02.079362 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:02.079341 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 18 00:46:02.085735 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:02.085710 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38527dde-928f-4086-b9ed-ef032d51cf48-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f477w5\" (UID: \"38527dde-928f-4086-b9ed-ef032d51cf48\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f477w5" Jul 18 00:46:02.085836 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:02.085742 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj2mp\" (UniqueName: \"kubernetes.io/projected/38527dde-928f-4086-b9ed-ef032d51cf48-kube-api-access-dj2mp\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f477w5\" (UID: \"38527dde-928f-4086-b9ed-ef032d51cf48\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f477w5" Jul 18 00:46:02.085836 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:02.085821 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38527dde-928f-4086-b9ed-ef032d51cf48-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f477w5\" (UID: \"38527dde-928f-4086-b9ed-ef032d51cf48\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f477w5" Jul 18 00:46:02.094746 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:02.094718 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qckh7"] Jul 18 00:46:02.186820 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:02.186778 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18731e25-f23c-4d26-8289-be3b22c45d45-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qckh7\" (UID: \"18731e25-f23c-4d26-8289-be3b22c45d45\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qckh7" Jul 18 00:46:02.186820 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:02.186830 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38527dde-928f-4086-b9ed-ef032d51cf48-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f477w5\" (UID: \"38527dde-928f-4086-b9ed-ef032d51cf48\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f477w5" Jul 18 00:46:02.187081 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:02.186861 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b44wz\" (UniqueName: \"kubernetes.io/projected/18731e25-f23c-4d26-8289-be3b22c45d45-kube-api-access-b44wz\") pod \"cert-manager-webhook-6486ffd997-qckh7\" (UID: \"18731e25-f23c-4d26-8289-be3b22c45d45\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qckh7" Jul 18 00:46:02.187081 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:02.186894 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38527dde-928f-4086-b9ed-ef032d51cf48-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f477w5\" (UID: \"38527dde-928f-4086-b9ed-ef032d51cf48\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f477w5" Jul 18 00:46:02.187081 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:02.186912 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj2mp\" (UniqueName: \"kubernetes.io/projected/38527dde-928f-4086-b9ed-ef032d51cf48-kube-api-access-dj2mp\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f477w5\" (UID: \"38527dde-928f-4086-b9ed-ef032d51cf48\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f477w5" Jul 18 00:46:02.187264 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:02.187242 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38527dde-928f-4086-b9ed-ef032d51cf48-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f477w5\" (UID: \"38527dde-928f-4086-b9ed-ef032d51cf48\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f477w5" Jul 18 00:46:02.187327 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:02.187249 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38527dde-928f-4086-b9ed-ef032d51cf48-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f477w5\" (UID: \"38527dde-928f-4086-b9ed-ef032d51cf48\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f477w5" Jul 18 00:46:02.195061 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:02.195038 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj2mp\" (UniqueName: \"kubernetes.io/projected/38527dde-928f-4086-b9ed-ef032d51cf48-kube-api-access-dj2mp\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f477w5\" (UID: \"38527dde-928f-4086-b9ed-ef032d51cf48\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f477w5" Jul 18 00:46:02.282882 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:02.282779 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f477w5" Jul 18 00:46:02.288415 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:02.288384 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18731e25-f23c-4d26-8289-be3b22c45d45-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qckh7\" (UID: \"18731e25-f23c-4d26-8289-be3b22c45d45\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qckh7" Jul 18 00:46:02.288619 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:02.288594 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b44wz\" (UniqueName: \"kubernetes.io/projected/18731e25-f23c-4d26-8289-be3b22c45d45-kube-api-access-b44wz\") pod \"cert-manager-webhook-6486ffd997-qckh7\" (UID: \"18731e25-f23c-4d26-8289-be3b22c45d45\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qckh7" Jul 18 00:46:02.297484 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:02.297446 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18731e25-f23c-4d26-8289-be3b22c45d45-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qckh7\" (UID: \"18731e25-f23c-4d26-8289-be3b22c45d45\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qckh7" Jul 18 00:46:02.297484 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:02.297473 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b44wz\" (UniqueName: \"kubernetes.io/projected/18731e25-f23c-4d26-8289-be3b22c45d45-kube-api-access-b44wz\") pod \"cert-manager-webhook-6486ffd997-qckh7\" (UID: \"18731e25-f23c-4d26-8289-be3b22c45d45\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qckh7" Jul 18 00:46:02.401983 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:02.401952 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-qckh7" Jul 18 00:46:02.415261 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:02.415237 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f477w5"] Jul 18 00:46:02.417698 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:46:02.417627 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38527dde_928f_4086_b9ed_ef032d51cf48.slice/crio-f515f1dae5f1d4ac441bbeb34d65eb9aac6510b6e97b9c50ed51c5440fd8d3cf WatchSource:0}: Error finding container f515f1dae5f1d4ac441bbeb34d65eb9aac6510b6e97b9c50ed51c5440fd8d3cf: Status 404 returned error can't find the container with id f515f1dae5f1d4ac441bbeb34d65eb9aac6510b6e97b9c50ed51c5440fd8d3cf Jul 18 00:46:02.521274 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:02.521237 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f477w5" event={"ID":"38527dde-928f-4086-b9ed-ef032d51cf48","Type":"ContainerStarted","Data":"dda40e0d8fe797e8366a8a3fcbe62804ea2906d8f312f7553bc97e054e8cec60"} Jul 18 00:46:02.521429 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:02.521293 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f477w5" event={"ID":"38527dde-928f-4086-b9ed-ef032d51cf48","Type":"ContainerStarted","Data":"f515f1dae5f1d4ac441bbeb34d65eb9aac6510b6e97b9c50ed51c5440fd8d3cf"} Jul 18 00:46:02.536084 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:02.536059 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qckh7"] Jul 18 00:46:02.560515 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:46:02.560479 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18731e25_f23c_4d26_8289_be3b22c45d45.slice/crio-5392318bd22e858f135cae01b98ce9c4d173058c4892a49d58c00377993875fd WatchSource:0}: Error finding container 5392318bd22e858f135cae01b98ce9c4d173058c4892a49d58c00377993875fd: Status 404 returned error can't find the container with id 5392318bd22e858f135cae01b98ce9c4d173058c4892a49d58c00377993875fd Jul 18 00:46:03.527405 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:03.527368 2580 generic.go:356] "Generic (PLEG): container finished" podID="38527dde-928f-4086-b9ed-ef032d51cf48" containerID="dda40e0d8fe797e8366a8a3fcbe62804ea2906d8f312f7553bc97e054e8cec60" exitCode=0 Jul 18 00:46:03.527955 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:03.527469 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f477w5" event={"ID":"38527dde-928f-4086-b9ed-ef032d51cf48","Type":"ContainerDied","Data":"dda40e0d8fe797e8366a8a3fcbe62804ea2906d8f312f7553bc97e054e8cec60"} Jul 18 00:46:03.528898 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:03.528864 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-qckh7" event={"ID":"18731e25-f23c-4d26-8289-be3b22c45d45","Type":"ContainerStarted","Data":"5392318bd22e858f135cae01b98ce9c4d173058c4892a49d58c00377993875fd"} Jul 18 00:46:06.547656 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:06.547625 2580 generic.go:356] "Generic (PLEG): container finished" podID="38527dde-928f-4086-b9ed-ef032d51cf48" containerID="1f57d29b91d27373af7a35a9a5b7305b04f4977c51ece4fa5f1bb37ef75dd876" exitCode=0 Jul 18 00:46:06.548131 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:06.547714 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f477w5" event={"ID":"38527dde-928f-4086-b9ed-ef032d51cf48","Type":"ContainerDied","Data":"1f57d29b91d27373af7a35a9a5b7305b04f4977c51ece4fa5f1bb37ef75dd876"} Jul 18 00:46:06.549374 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:06.549348 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-qckh7" event={"ID":"18731e25-f23c-4d26-8289-be3b22c45d45","Type":"ContainerStarted","Data":"58534bd9ca1631e5c9cb6801e09f65d025f9f41a8e4e2ecf38f53f20cc50012e"} Jul 18 00:46:06.549472 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:06.549394 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-qckh7" Jul 18 00:46:06.584423 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:06.584370 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-qckh7" podStartSLOduration=1.253594464 podStartE2EDuration="4.584352787s" totalImagesPullingTime="3.330758323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:46:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:46:06.583496841 +0000 UTC m=+326.935347187" watchObservedRunningTime="2026-07-18 00:46:06.584352787 +0000 UTC m=+326.936203131" Jul 18 00:46:07.555536 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:07.555498 2580 generic.go:356] "Generic (PLEG): container finished" podID="38527dde-928f-4086-b9ed-ef032d51cf48" containerID="a32c09aea93cc480c0ef93c3e6b24ef980ed830d478c882227be412437b1651a" exitCode=0 Jul 18 00:46:07.555992 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:07.555590 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f477w5" event={"ID":"38527dde-928f-4086-b9ed-ef032d51cf48","Type":"ContainerDied","Data":"a32c09aea93cc480c0ef93c3e6b24ef980ed830d478c882227be412437b1651a"} Jul 18 00:46:08.712173 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:08.712148 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f477w5" Jul 18 00:46:08.842954 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:08.842862 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38527dde-928f-4086-b9ed-ef032d51cf48-util\") pod \"38527dde-928f-4086-b9ed-ef032d51cf48\" (UID: \"38527dde-928f-4086-b9ed-ef032d51cf48\") " Jul 18 00:46:08.842954 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:08.842909 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dj2mp\" (UniqueName: \"kubernetes.io/projected/38527dde-928f-4086-b9ed-ef032d51cf48-kube-api-access-dj2mp\") pod \"38527dde-928f-4086-b9ed-ef032d51cf48\" (UID: \"38527dde-928f-4086-b9ed-ef032d51cf48\") " Jul 18 00:46:08.843184 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:08.842989 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38527dde-928f-4086-b9ed-ef032d51cf48-bundle\") pod \"38527dde-928f-4086-b9ed-ef032d51cf48\" (UID: \"38527dde-928f-4086-b9ed-ef032d51cf48\") " Jul 18 00:46:08.843379 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:08.843356 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38527dde-928f-4086-b9ed-ef032d51cf48-bundle" (OuterVolumeSpecName: "bundle") pod "38527dde-928f-4086-b9ed-ef032d51cf48" (UID: "38527dde-928f-4086-b9ed-ef032d51cf48"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 00:46:08.845212 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:08.845187 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38527dde-928f-4086-b9ed-ef032d51cf48-kube-api-access-dj2mp" (OuterVolumeSpecName: "kube-api-access-dj2mp") pod "38527dde-928f-4086-b9ed-ef032d51cf48" (UID: "38527dde-928f-4086-b9ed-ef032d51cf48"). InnerVolumeSpecName "kube-api-access-dj2mp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 00:46:08.846998 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:08.846963 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38527dde-928f-4086-b9ed-ef032d51cf48-util" (OuterVolumeSpecName: "util") pod "38527dde-928f-4086-b9ed-ef032d51cf48" (UID: "38527dde-928f-4086-b9ed-ef032d51cf48"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 00:46:08.859034 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:08.859008 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mm5q8"] Jul 18 00:46:08.859274 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:08.859264 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38527dde-928f-4086-b9ed-ef032d51cf48" containerName="pull" Jul 18 00:46:08.859314 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:08.859276 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="38527dde-928f-4086-b9ed-ef032d51cf48" containerName="pull" Jul 18 00:46:08.859314 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:08.859285 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38527dde-928f-4086-b9ed-ef032d51cf48" containerName="extract" Jul 18 00:46:08.859314 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:08.859290 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="38527dde-928f-4086-b9ed-ef032d51cf48" containerName="extract" Jul 18 00:46:08.859314 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:08.859300 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38527dde-928f-4086-b9ed-ef032d51cf48" containerName="util" Jul 18 00:46:08.859314 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:08.859305 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="38527dde-928f-4086-b9ed-ef032d51cf48" containerName="util" Jul 18 00:46:08.859470 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:08.859344 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="38527dde-928f-4086-b9ed-ef032d51cf48" containerName="extract" Jul 18 00:46:08.861604 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:08.861587 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mm5q8" Jul 18 00:46:08.864207 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:08.864178 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-kgh6c\"" Jul 18 00:46:08.869855 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:08.869828 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mm5q8"] Jul 18 00:46:08.943455 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:08.943418 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38527dde-928f-4086-b9ed-ef032d51cf48-bundle\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:46:08.943455 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:08.943447 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38527dde-928f-4086-b9ed-ef032d51cf48-util\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:46:08.943455 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:08.943455 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj2mp\" (UniqueName: \"kubernetes.io/projected/38527dde-928f-4086-b9ed-ef032d51cf48-kube-api-access-dj2mp\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:46:09.044281 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:09.044243 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnw2t\" (UniqueName: \"kubernetes.io/projected/d750721a-1dcb-4572-9e17-ff5a46a3a341-kube-api-access-tnw2t\") pod \"cert-manager-cainjector-559cdd8c87-mm5q8\" (UID: \"d750721a-1dcb-4572-9e17-ff5a46a3a341\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mm5q8" Jul 18 00:46:09.044438 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:09.044308 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d750721a-1dcb-4572-9e17-ff5a46a3a341-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mm5q8\" (UID: \"d750721a-1dcb-4572-9e17-ff5a46a3a341\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mm5q8" Jul 18 00:46:09.145123 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:09.145051 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnw2t\" (UniqueName: \"kubernetes.io/projected/d750721a-1dcb-4572-9e17-ff5a46a3a341-kube-api-access-tnw2t\") pod \"cert-manager-cainjector-559cdd8c87-mm5q8\" (UID: \"d750721a-1dcb-4572-9e17-ff5a46a3a341\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mm5q8" Jul 18 00:46:09.145123 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:09.145106 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d750721a-1dcb-4572-9e17-ff5a46a3a341-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mm5q8\" (UID: \"d750721a-1dcb-4572-9e17-ff5a46a3a341\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mm5q8" Jul 18 00:46:09.153016 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:09.152974 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d750721a-1dcb-4572-9e17-ff5a46a3a341-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mm5q8\" (UID: \"d750721a-1dcb-4572-9e17-ff5a46a3a341\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mm5q8" Jul 18 00:46:09.153233 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:09.153211 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnw2t\" (UniqueName: \"kubernetes.io/projected/d750721a-1dcb-4572-9e17-ff5a46a3a341-kube-api-access-tnw2t\") pod \"cert-manager-cainjector-559cdd8c87-mm5q8\" (UID: \"d750721a-1dcb-4572-9e17-ff5a46a3a341\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mm5q8" Jul 18 00:46:09.170979 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:09.170953 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mm5q8" Jul 18 00:46:09.296877 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:09.296854 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mm5q8"] Jul 18 00:46:09.299207 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:46:09.299179 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd750721a_1dcb_4572_9e17_ff5a46a3a341.slice/crio-4c009b2de9adf3cb0d457cef72c85fc9270f297016b93362a51ac5a0083a5a53 WatchSource:0}: Error finding container 4c009b2de9adf3cb0d457cef72c85fc9270f297016b93362a51ac5a0083a5a53: Status 404 returned error can't find the container with id 4c009b2de9adf3cb0d457cef72c85fc9270f297016b93362a51ac5a0083a5a53 Jul 18 00:46:09.565844 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:09.565805 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mm5q8" event={"ID":"d750721a-1dcb-4572-9e17-ff5a46a3a341","Type":"ContainerStarted","Data":"f2df1c624d0c23751949fe244cde99117e53876b89a79b8692ff1e1a0fae29c1"} Jul 18 00:46:09.566055 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:09.565855 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mm5q8" event={"ID":"d750721a-1dcb-4572-9e17-ff5a46a3a341","Type":"ContainerStarted","Data":"4c009b2de9adf3cb0d457cef72c85fc9270f297016b93362a51ac5a0083a5a53"} Jul 18 00:46:09.567874 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:09.567850 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f477w5" Jul 18 00:46:09.568006 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:09.567849 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f477w5" event={"ID":"38527dde-928f-4086-b9ed-ef032d51cf48","Type":"ContainerDied","Data":"f515f1dae5f1d4ac441bbeb34d65eb9aac6510b6e97b9c50ed51c5440fd8d3cf"} Jul 18 00:46:09.568006 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:09.567984 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f515f1dae5f1d4ac441bbeb34d65eb9aac6510b6e97b9c50ed51c5440fd8d3cf" Jul 18 00:46:09.581630 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:09.581585 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mm5q8" podStartSLOduration=1.581569692 podStartE2EDuration="1.581569692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:46:09.580943863 +0000 UTC m=+329.932794211" watchObservedRunningTime="2026-07-18 00:46:09.581569692 +0000 UTC m=+329.933420036" Jul 18 00:46:12.558330 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:12.558287 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-qckh7" Jul 18 00:46:20.063210 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:20.063175 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xx8q4"] Jul 18 00:46:20.065261 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:20.065243 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xx8q4" Jul 18 00:46:20.067875 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:20.067849 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 18 00:46:20.068003 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:20.067860 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 18 00:46:20.068730 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:20.068715 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-dzz8k\"" Jul 18 00:46:20.075131 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:20.075106 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xx8q4"] Jul 18 00:46:20.138976 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:20.138934 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/777066f8-7c6b-4be1-b328-224d160dcd29-tmp\") pod \"openshift-lws-operator-fd8ccff4c-xx8q4\" (UID: \"777066f8-7c6b-4be1-b328-224d160dcd29\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xx8q4" Jul 18 00:46:20.139147 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:20.139021 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9f4b\" (UniqueName: \"kubernetes.io/projected/777066f8-7c6b-4be1-b328-224d160dcd29-kube-api-access-h9f4b\") pod \"openshift-lws-operator-fd8ccff4c-xx8q4\" (UID: \"777066f8-7c6b-4be1-b328-224d160dcd29\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xx8q4" Jul 18 00:46:20.239962 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:20.239918 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9f4b\" (UniqueName: \"kubernetes.io/projected/777066f8-7c6b-4be1-b328-224d160dcd29-kube-api-access-h9f4b\") pod \"openshift-lws-operator-fd8ccff4c-xx8q4\" (UID: \"777066f8-7c6b-4be1-b328-224d160dcd29\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xx8q4" Jul 18 00:46:20.239962 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:20.239973 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/777066f8-7c6b-4be1-b328-224d160dcd29-tmp\") pod \"openshift-lws-operator-fd8ccff4c-xx8q4\" (UID: \"777066f8-7c6b-4be1-b328-224d160dcd29\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xx8q4" Jul 18 00:46:20.240312 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:20.240296 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/777066f8-7c6b-4be1-b328-224d160dcd29-tmp\") pod \"openshift-lws-operator-fd8ccff4c-xx8q4\" (UID: \"777066f8-7c6b-4be1-b328-224d160dcd29\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xx8q4" Jul 18 00:46:20.247703 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:20.247655 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9f4b\" (UniqueName: \"kubernetes.io/projected/777066f8-7c6b-4be1-b328-224d160dcd29-kube-api-access-h9f4b\") pod \"openshift-lws-operator-fd8ccff4c-xx8q4\" (UID: \"777066f8-7c6b-4be1-b328-224d160dcd29\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xx8q4" Jul 18 00:46:20.374651 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:20.374556 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xx8q4" Jul 18 00:46:20.508834 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:20.508761 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xx8q4"] Jul 18 00:46:20.511568 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:46:20.511542 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod777066f8_7c6b_4be1_b328_224d160dcd29.slice/crio-5d25d722f99529a70e5f9c5fdd53d2b6f6695be60a296e74a06052846f049dcc WatchSource:0}: Error finding container 5d25d722f99529a70e5f9c5fdd53d2b6f6695be60a296e74a06052846f049dcc: Status 404 returned error can't find the container with id 5d25d722f99529a70e5f9c5fdd53d2b6f6695be60a296e74a06052846f049dcc Jul 18 00:46:20.612101 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:20.612064 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xx8q4" event={"ID":"777066f8-7c6b-4be1-b328-224d160dcd29","Type":"ContainerStarted","Data":"5d25d722f99529a70e5f9c5fdd53d2b6f6695be60a296e74a06052846f049dcc"} Jul 18 00:46:22.623335 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:22.623300 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xx8q4" event={"ID":"777066f8-7c6b-4be1-b328-224d160dcd29","Type":"ContainerStarted","Data":"9d1c7a04849e3258fb7202b5802eb206218b9e5ae938132110f4fd58ce95ef04"} Jul 18 00:46:22.640056 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:22.640008 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xx8q4" podStartSLOduration=1.140488221 podStartE2EDuration="2.639992464s" totalImagesPullingTime="1.499504243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:46:22.638693787 +0000 UTC m=+342.990544136" watchObservedRunningTime="2026-07-18 00:46:22.639992464 +0000 UTC m=+342.991842809" Jul 18 00:46:25.201614 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:25.201571 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqnwwd"] Jul 18 00:46:25.218874 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:25.218840 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqnwwd"] Jul 18 00:46:25.218874 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:25.218850 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqnwwd" Jul 18 00:46:25.221406 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:25.221386 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 18 00:46:25.222368 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:25.222350 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9rsk7\"" Jul 18 00:46:25.222647 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:25.222366 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 18 00:46:25.275912 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:25.275869 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d7d6880-bca7-42fb-8c4a-92b86eb2d782-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqnwwd\" (UID: \"4d7d6880-bca7-42fb-8c4a-92b86eb2d782\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqnwwd" Jul 18 00:46:25.276086 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:25.275928 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rvnx\" (UniqueName: \"kubernetes.io/projected/4d7d6880-bca7-42fb-8c4a-92b86eb2d782-kube-api-access-4rvnx\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqnwwd\" (UID: \"4d7d6880-bca7-42fb-8c4a-92b86eb2d782\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqnwwd" Jul 18 00:46:25.276086 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:25.275977 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d7d6880-bca7-42fb-8c4a-92b86eb2d782-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqnwwd\" (UID: \"4d7d6880-bca7-42fb-8c4a-92b86eb2d782\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqnwwd" Jul 18 00:46:25.377230 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:25.377187 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rvnx\" (UniqueName: \"kubernetes.io/projected/4d7d6880-bca7-42fb-8c4a-92b86eb2d782-kube-api-access-4rvnx\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqnwwd\" (UID: \"4d7d6880-bca7-42fb-8c4a-92b86eb2d782\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqnwwd" Jul 18 00:46:25.377230 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:25.377238 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d7d6880-bca7-42fb-8c4a-92b86eb2d782-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqnwwd\" (UID: \"4d7d6880-bca7-42fb-8c4a-92b86eb2d782\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqnwwd" Jul 18 00:46:25.377476 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:25.377298 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d7d6880-bca7-42fb-8c4a-92b86eb2d782-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqnwwd\" (UID: \"4d7d6880-bca7-42fb-8c4a-92b86eb2d782\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqnwwd" Jul 18 00:46:25.377717 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:25.377695 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d7d6880-bca7-42fb-8c4a-92b86eb2d782-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqnwwd\" (UID: \"4d7d6880-bca7-42fb-8c4a-92b86eb2d782\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqnwwd" Jul 18 00:46:25.377786 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:25.377713 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d7d6880-bca7-42fb-8c4a-92b86eb2d782-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqnwwd\" (UID: \"4d7d6880-bca7-42fb-8c4a-92b86eb2d782\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqnwwd" Jul 18 00:46:25.385639 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:25.385615 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rvnx\" (UniqueName: \"kubernetes.io/projected/4d7d6880-bca7-42fb-8c4a-92b86eb2d782-kube-api-access-4rvnx\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqnwwd\" (UID: \"4d7d6880-bca7-42fb-8c4a-92b86eb2d782\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqnwwd" Jul 18 00:46:25.528129 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:25.528093 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqnwwd" Jul 18 00:46:25.676007 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:25.675980 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqnwwd"] Jul 18 00:46:25.678087 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:46:25.678063 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7d6880_bca7_42fb_8c4a_92b86eb2d782.slice/crio-833053c78d56c09b3555ccd3c3b903740c507489ab883d4e09ddfab05bbd13e6 WatchSource:0}: Error finding container 833053c78d56c09b3555ccd3c3b903740c507489ab883d4e09ddfab05bbd13e6: Status 404 returned error can't find the container with id 833053c78d56c09b3555ccd3c3b903740c507489ab883d4e09ddfab05bbd13e6 Jul 18 00:46:26.642356 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:26.642248 2580 generic.go:356] "Generic (PLEG): container finished" podID="4d7d6880-bca7-42fb-8c4a-92b86eb2d782" containerID="3483e90c644a63029c273309790ed025e4ff4d2b2ad1d70b97d3713bf755015e" exitCode=0 Jul 18 00:46:26.642356 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:26.642313 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqnwwd" event={"ID":"4d7d6880-bca7-42fb-8c4a-92b86eb2d782","Type":"ContainerDied","Data":"3483e90c644a63029c273309790ed025e4ff4d2b2ad1d70b97d3713bf755015e"} Jul 18 00:46:26.642356 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:26.642356 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqnwwd" event={"ID":"4d7d6880-bca7-42fb-8c4a-92b86eb2d782","Type":"ContainerStarted","Data":"833053c78d56c09b3555ccd3c3b903740c507489ab883d4e09ddfab05bbd13e6"} Jul 18 00:46:27.647918 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:27.647885 2580 generic.go:356] "Generic (PLEG): container finished" podID="4d7d6880-bca7-42fb-8c4a-92b86eb2d782" containerID="dd4c21f9f31bb44b4ebd2ad4a047b0cfd33f762470fe1502614c4759b438bacf" exitCode=0 Jul 18 00:46:27.648322 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:27.647983 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqnwwd" event={"ID":"4d7d6880-bca7-42fb-8c4a-92b86eb2d782","Type":"ContainerDied","Data":"dd4c21f9f31bb44b4ebd2ad4a047b0cfd33f762470fe1502614c4759b438bacf"} Jul 18 00:46:28.654562 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:28.654527 2580 generic.go:356] "Generic (PLEG): container finished" podID="4d7d6880-bca7-42fb-8c4a-92b86eb2d782" containerID="fa167e9ae88d7f771bc5e7592536593c55801df328a3158e7b2e1a703222d40a" exitCode=0 Jul 18 00:46:28.654562 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:28.654564 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqnwwd" event={"ID":"4d7d6880-bca7-42fb-8c4a-92b86eb2d782","Type":"ContainerDied","Data":"fa167e9ae88d7f771bc5e7592536593c55801df328a3158e7b2e1a703222d40a"} Jul 18 00:46:29.789083 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:29.789059 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqnwwd" Jul 18 00:46:29.815367 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:29.815336 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d7d6880-bca7-42fb-8c4a-92b86eb2d782-util\") pod \"4d7d6880-bca7-42fb-8c4a-92b86eb2d782\" (UID: \"4d7d6880-bca7-42fb-8c4a-92b86eb2d782\") " Jul 18 00:46:29.815534 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:29.815381 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d7d6880-bca7-42fb-8c4a-92b86eb2d782-bundle\") pod \"4d7d6880-bca7-42fb-8c4a-92b86eb2d782\" (UID: \"4d7d6880-bca7-42fb-8c4a-92b86eb2d782\") " Jul 18 00:46:29.815534 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:29.815406 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rvnx\" (UniqueName: \"kubernetes.io/projected/4d7d6880-bca7-42fb-8c4a-92b86eb2d782-kube-api-access-4rvnx\") pod \"4d7d6880-bca7-42fb-8c4a-92b86eb2d782\" (UID: \"4d7d6880-bca7-42fb-8c4a-92b86eb2d782\") " Jul 18 00:46:29.816323 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:29.816289 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d7d6880-bca7-42fb-8c4a-92b86eb2d782-bundle" (OuterVolumeSpecName: "bundle") pod "4d7d6880-bca7-42fb-8c4a-92b86eb2d782" (UID: "4d7d6880-bca7-42fb-8c4a-92b86eb2d782"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 00:46:29.817612 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:29.817592 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d7d6880-bca7-42fb-8c4a-92b86eb2d782-kube-api-access-4rvnx" (OuterVolumeSpecName: "kube-api-access-4rvnx") pod "4d7d6880-bca7-42fb-8c4a-92b86eb2d782" (UID: "4d7d6880-bca7-42fb-8c4a-92b86eb2d782"). InnerVolumeSpecName "kube-api-access-4rvnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 00:46:29.823190 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:29.823144 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d7d6880-bca7-42fb-8c4a-92b86eb2d782-util" (OuterVolumeSpecName: "util") pod "4d7d6880-bca7-42fb-8c4a-92b86eb2d782" (UID: "4d7d6880-bca7-42fb-8c4a-92b86eb2d782"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 00:46:29.916122 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:29.916040 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d7d6880-bca7-42fb-8c4a-92b86eb2d782-util\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:46:29.916122 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:29.916068 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d7d6880-bca7-42fb-8c4a-92b86eb2d782-bundle\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:46:29.916122 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:29.916078 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rvnx\" (UniqueName: \"kubernetes.io/projected/4d7d6880-bca7-42fb-8c4a-92b86eb2d782-kube-api-access-4rvnx\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:46:30.665399 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:30.665317 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqnwwd" event={"ID":"4d7d6880-bca7-42fb-8c4a-92b86eb2d782","Type":"ContainerDied","Data":"833053c78d56c09b3555ccd3c3b903740c507489ab883d4e09ddfab05bbd13e6"} Jul 18 00:46:30.665399 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:30.665375 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="833053c78d56c09b3555ccd3c3b903740c507489ab883d4e09ddfab05bbd13e6" Jul 18 00:46:30.665399 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:30.665330 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqnwwd" Jul 18 00:46:39.146150 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.146119 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w"] Jul 18 00:46:39.146523 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.146388 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d7d6880-bca7-42fb-8c4a-92b86eb2d782" containerName="extract" Jul 18 00:46:39.146523 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.146399 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d7d6880-bca7-42fb-8c4a-92b86eb2d782" containerName="extract" Jul 18 00:46:39.146523 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.146412 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d7d6880-bca7-42fb-8c4a-92b86eb2d782" containerName="pull" Jul 18 00:46:39.146523 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.146418 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d7d6880-bca7-42fb-8c4a-92b86eb2d782" containerName="pull" Jul 18 00:46:39.146523 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.146425 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d7d6880-bca7-42fb-8c4a-92b86eb2d782" containerName="util" Jul 18 00:46:39.146523 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.146430 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d7d6880-bca7-42fb-8c4a-92b86eb2d782" containerName="util" Jul 18 00:46:39.146523 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.146477 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d7d6880-bca7-42fb-8c4a-92b86eb2d782" containerName="extract" Jul 18 00:46:39.189215 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.189187 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w"] Jul 18 00:46:39.189368 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.189286 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w" Jul 18 00:46:39.196140 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.196117 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 18 00:46:39.196627 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.196610 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jk4rp\"" Jul 18 00:46:39.196923 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.196910 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 18 00:46:39.197202 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.197181 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 18 00:46:39.201659 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.201642 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 18 00:46:39.292250 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.292210 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92-manifests\") pod \"opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w\" (UID: \"f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92\") " pod="opendatahub/opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w" Jul 18 00:46:39.292250 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.292250 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w\" (UID: \"f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92\") " pod="opendatahub/opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w" Jul 18 00:46:39.292479 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.292272 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92-tmp\") pod \"opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w\" (UID: \"f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92\") " pod="opendatahub/opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w" Jul 18 00:46:39.292479 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.292289 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w\" (UID: \"f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92\") " pod="opendatahub/opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w" Jul 18 00:46:39.292479 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.292310 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74749\" (UniqueName: \"kubernetes.io/projected/f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92-kube-api-access-74749\") pod \"opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w\" (UID: \"f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92\") " pod="opendatahub/opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w" Jul 18 00:46:39.393488 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.393447 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92-manifests\") pod \"opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w\" (UID: \"f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92\") " pod="opendatahub/opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w" Jul 18 00:46:39.393488 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.393492 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w\" (UID: \"f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92\") " pod="opendatahub/opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w" Jul 18 00:46:39.393739 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.393515 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92-tmp\") pod \"opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w\" (UID: \"f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92\") " pod="opendatahub/opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w" Jul 18 00:46:39.393739 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.393628 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w\" (UID: \"f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92\") " pod="opendatahub/opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w" Jul 18 00:46:39.393739 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.393703 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74749\" (UniqueName: \"kubernetes.io/projected/f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92-kube-api-access-74749\") pod \"opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w\" (UID: \"f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92\") " pod="opendatahub/opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w" Jul 18 00:46:39.394011 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.393988 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92-manifests\") pod \"opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w\" (UID: \"f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92\") " pod="opendatahub/opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w" Jul 18 00:46:39.396058 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.396033 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92-tmp\") pod \"opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w\" (UID: \"f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92\") " pod="opendatahub/opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w" Jul 18 00:46:39.396186 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.396130 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w\" (UID: \"f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92\") " pod="opendatahub/opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w" Jul 18 00:46:39.396257 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.396241 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w\" (UID: \"f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92\") " pod="opendatahub/opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w" Jul 18 00:46:39.414763 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.414732 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74749\" (UniqueName: \"kubernetes.io/projected/f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92-kube-api-access-74749\") pod \"opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w\" (UID: \"f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92\") " pod="opendatahub/opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w" Jul 18 00:46:39.499222 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.499187 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w" Jul 18 00:46:39.667889 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.667811 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w"] Jul 18 00:46:39.670252 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:46:39.670228 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3efa0cb_1b7c_44ca_a31b_fd6de43eaa92.slice/crio-7ae490230b8a559f332c7a8f7f48192ffe566f27cf0e2448b9c45aa2e3eb4eae WatchSource:0}: Error finding container 7ae490230b8a559f332c7a8f7f48192ffe566f27cf0e2448b9c45aa2e3eb4eae: Status 404 returned error can't find the container with id 7ae490230b8a559f332c7a8f7f48192ffe566f27cf0e2448b9c45aa2e3eb4eae Jul 18 00:46:39.706538 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:39.706498 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w" event={"ID":"f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92","Type":"ContainerStarted","Data":"7ae490230b8a559f332c7a8f7f48192ffe566f27cf0e2448b9c45aa2e3eb4eae"} Jul 18 00:46:42.723653 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:42.723613 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w" event={"ID":"f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92","Type":"ContainerStarted","Data":"8fc5946729f7502c0e3d9c388c38c8615d2d5938b02cc216da5349315b25b450"} Jul 18 00:46:43.729125 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:43.729092 2580 generic.go:356] "Generic (PLEG): container finished" podID="f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92" containerID="8fc5946729f7502c0e3d9c388c38c8615d2d5938b02cc216da5349315b25b450" exitCode=0 Jul 18 00:46:43.729510 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:43.729139 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w" event={"ID":"f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92","Type":"ContainerDied","Data":"8fc5946729f7502c0e3d9c388c38c8615d2d5938b02cc216da5349315b25b450"} Jul 18 00:46:44.735746 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:44.735705 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w" event={"ID":"f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92","Type":"ContainerStarted","Data":"16369ca4f1378767445fd971453d627ec89c76213827ecf02a0ee3b4de9044c8"} Jul 18 00:46:44.736210 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:44.735874 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w" Jul 18 00:46:44.760226 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:44.760180 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w" podStartSLOduration=1.446743876 podStartE2EDuration="5.760164663s" totalImagesPullingTime="3.226873285s" totalInitContainerRuntime="1.086547502s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:46:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:46:44.758390689 +0000 UTC m=+365.110241034" watchObservedRunningTime="2026-07-18 00:46:44.760164663 +0000 UTC m=+365.112015008" Jul 18 00:46:55.743733 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:55.743694 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w" Jul 18 00:46:56.138443 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:56.138351 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-845f554b8f-989s7"] Jul 18 00:46:56.148292 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:56.148262 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-845f554b8f-989s7" Jul 18 00:46:56.152649 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:56.152615 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 18 00:46:56.153298 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:56.153269 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 18 00:46:56.153775 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:56.153749 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 18 00:46:56.156061 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:56.153880 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-j8jws\"" Jul 18 00:46:56.161754 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:56.161390 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-845f554b8f-989s7"] Jul 18 00:46:56.231893 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:56.231857 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/63cbaf43-e409-4e8f-8280-aed86a0594cc-metrics-cert\") pod \"lws-controller-manager-845f554b8f-989s7\" (UID: \"63cbaf43-e409-4e8f-8280-aed86a0594cc\") " pod="openshift-lws-operator/lws-controller-manager-845f554b8f-989s7" Jul 18 00:46:56.231893 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:56.231900 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd4cs\" (UniqueName: \"kubernetes.io/projected/63cbaf43-e409-4e8f-8280-aed86a0594cc-kube-api-access-qd4cs\") pod \"lws-controller-manager-845f554b8f-989s7\" (UID: \"63cbaf43-e409-4e8f-8280-aed86a0594cc\") " pod="openshift-lws-operator/lws-controller-manager-845f554b8f-989s7" Jul 18 00:46:56.232111 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:56.231924 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63cbaf43-e409-4e8f-8280-aed86a0594cc-cert\") pod \"lws-controller-manager-845f554b8f-989s7\" (UID: \"63cbaf43-e409-4e8f-8280-aed86a0594cc\") " pod="openshift-lws-operator/lws-controller-manager-845f554b8f-989s7" Jul 18 00:46:56.232111 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:56.231998 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/63cbaf43-e409-4e8f-8280-aed86a0594cc-manager-config\") pod \"lws-controller-manager-845f554b8f-989s7\" (UID: \"63cbaf43-e409-4e8f-8280-aed86a0594cc\") " pod="openshift-lws-operator/lws-controller-manager-845f554b8f-989s7" Jul 18 00:46:56.333039 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:56.333002 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/63cbaf43-e409-4e8f-8280-aed86a0594cc-metrics-cert\") pod \"lws-controller-manager-845f554b8f-989s7\" (UID: \"63cbaf43-e409-4e8f-8280-aed86a0594cc\") " pod="openshift-lws-operator/lws-controller-manager-845f554b8f-989s7" Jul 18 00:46:56.333039 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:56.333045 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd4cs\" (UniqueName: \"kubernetes.io/projected/63cbaf43-e409-4e8f-8280-aed86a0594cc-kube-api-access-qd4cs\") pod \"lws-controller-manager-845f554b8f-989s7\" (UID: \"63cbaf43-e409-4e8f-8280-aed86a0594cc\") " pod="openshift-lws-operator/lws-controller-manager-845f554b8f-989s7" Jul 18 00:46:56.333242 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:56.333077 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63cbaf43-e409-4e8f-8280-aed86a0594cc-cert\") pod \"lws-controller-manager-845f554b8f-989s7\" (UID: \"63cbaf43-e409-4e8f-8280-aed86a0594cc\") " pod="openshift-lws-operator/lws-controller-manager-845f554b8f-989s7" Jul 18 00:46:56.333242 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:56.333108 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/63cbaf43-e409-4e8f-8280-aed86a0594cc-manager-config\") pod \"lws-controller-manager-845f554b8f-989s7\" (UID: \"63cbaf43-e409-4e8f-8280-aed86a0594cc\") " pod="openshift-lws-operator/lws-controller-manager-845f554b8f-989s7" Jul 18 00:46:56.333923 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:56.333827 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/63cbaf43-e409-4e8f-8280-aed86a0594cc-manager-config\") pod \"lws-controller-manager-845f554b8f-989s7\" (UID: \"63cbaf43-e409-4e8f-8280-aed86a0594cc\") " pod="openshift-lws-operator/lws-controller-manager-845f554b8f-989s7" Jul 18 00:46:56.335741 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:56.335711 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/63cbaf43-e409-4e8f-8280-aed86a0594cc-metrics-cert\") pod \"lws-controller-manager-845f554b8f-989s7\" (UID: \"63cbaf43-e409-4e8f-8280-aed86a0594cc\") " pod="openshift-lws-operator/lws-controller-manager-845f554b8f-989s7" Jul 18 00:46:56.335841 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:56.335758 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63cbaf43-e409-4e8f-8280-aed86a0594cc-cert\") pod \"lws-controller-manager-845f554b8f-989s7\" (UID: \"63cbaf43-e409-4e8f-8280-aed86a0594cc\") " pod="openshift-lws-operator/lws-controller-manager-845f554b8f-989s7" Jul 18 00:46:56.341614 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:56.341584 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd4cs\" (UniqueName: \"kubernetes.io/projected/63cbaf43-e409-4e8f-8280-aed86a0594cc-kube-api-access-qd4cs\") pod \"lws-controller-manager-845f554b8f-989s7\" (UID: \"63cbaf43-e409-4e8f-8280-aed86a0594cc\") " pod="openshift-lws-operator/lws-controller-manager-845f554b8f-989s7" Jul 18 00:46:56.464997 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:56.464942 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-845f554b8f-989s7" Jul 18 00:46:56.614202 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:56.614170 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-845f554b8f-989s7"] Jul 18 00:46:56.616852 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:46:56.616820 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63cbaf43_e409_4e8f_8280_aed86a0594cc.slice/crio-16196cf77f3879c69b55213f46bdd131432a6db0c49e9819d8960471f6709f58 WatchSource:0}: Error finding container 16196cf77f3879c69b55213f46bdd131432a6db0c49e9819d8960471f6709f58: Status 404 returned error can't find the container with id 16196cf77f3879c69b55213f46bdd131432a6db0c49e9819d8960471f6709f58 Jul 18 00:46:56.786822 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:56.786733 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-845f554b8f-989s7" event={"ID":"63cbaf43-e409-4e8f-8280-aed86a0594cc","Type":"ContainerStarted","Data":"16196cf77f3879c69b55213f46bdd131432a6db0c49e9819d8960471f6709f58"} Jul 18 00:46:58.796183 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:58.796137 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-845f554b8f-989s7" event={"ID":"63cbaf43-e409-4e8f-8280-aed86a0594cc","Type":"ContainerStarted","Data":"5f3ac3bcf7c1da97b5eb1c7fbfb477f131132034c9dcd0548faaaf9a1f508ce0"} Jul 18 00:46:58.796587 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:46:58.796224 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-845f554b8f-989s7" Jul 18 00:47:08.396307 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.396239 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-845f554b8f-989s7" podStartSLOduration=11.181216525 podStartE2EDuration="12.396214451s" totalImagesPullingTime="1.214997926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:46:58.824663552 +0000 UTC m=+379.176513898" watchObservedRunningTime="2026-07-18 00:47:08.396214451 +0000 UTC m=+388.748064833" Jul 18 00:47:08.397966 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.397931 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jg4w7"] Jul 18 00:47:08.401416 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.401394 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jg4w7" Jul 18 00:47:08.410774 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.410754 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 18 00:47:08.410855 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.410754 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 18 00:47:08.410910 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.410753 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 18 00:47:08.411072 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.411060 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 18 00:47:08.411121 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.411090 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 18 00:47:08.411198 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.411178 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-df2vt\"" Jul 18 00:47:08.412152 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.412140 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 18 00:47:08.430610 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.430539 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jg4w7"] Jul 18 00:47:08.541463 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.541434 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8efec2bf-6c93-4171-9363-772e969aa333-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-jg4w7\" (UID: \"8efec2bf-6c93-4171-9363-772e969aa333\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jg4w7" Jul 18 00:47:08.541618 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.541469 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8efec2bf-6c93-4171-9363-772e969aa333-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-jg4w7\" (UID: \"8efec2bf-6c93-4171-9363-772e969aa333\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jg4w7" Jul 18 00:47:08.541618 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.541552 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8efec2bf-6c93-4171-9363-772e969aa333-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-jg4w7\" (UID: \"8efec2bf-6c93-4171-9363-772e969aa333\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jg4w7" Jul 18 00:47:08.541618 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.541588 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8efec2bf-6c93-4171-9363-772e969aa333-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-jg4w7\" (UID: \"8efec2bf-6c93-4171-9363-772e969aa333\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jg4w7" Jul 18 00:47:08.541618 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.541607 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hsdb\" (UniqueName: \"kubernetes.io/projected/8efec2bf-6c93-4171-9363-772e969aa333-kube-api-access-5hsdb\") pod \"istiod-openshift-gateway-94bb8fbfd-jg4w7\" (UID: \"8efec2bf-6c93-4171-9363-772e969aa333\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jg4w7" Jul 18 00:47:08.541769 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.541626 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8efec2bf-6c93-4171-9363-772e969aa333-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-jg4w7\" (UID: \"8efec2bf-6c93-4171-9363-772e969aa333\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jg4w7" Jul 18 00:47:08.541769 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.541651 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8efec2bf-6c93-4171-9363-772e969aa333-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-jg4w7\" (UID: \"8efec2bf-6c93-4171-9363-772e969aa333\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jg4w7" Jul 18 00:47:08.642821 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.642786 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8efec2bf-6c93-4171-9363-772e969aa333-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-jg4w7\" (UID: \"8efec2bf-6c93-4171-9363-772e969aa333\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jg4w7" Jul 18 00:47:08.642821 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.642820 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8efec2bf-6c93-4171-9363-772e969aa333-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-jg4w7\" (UID: \"8efec2bf-6c93-4171-9363-772e969aa333\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jg4w7" Jul 18 00:47:08.643075 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.642843 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hsdb\" (UniqueName: \"kubernetes.io/projected/8efec2bf-6c93-4171-9363-772e969aa333-kube-api-access-5hsdb\") pod \"istiod-openshift-gateway-94bb8fbfd-jg4w7\" (UID: \"8efec2bf-6c93-4171-9363-772e969aa333\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jg4w7" Jul 18 00:47:08.643075 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.642964 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8efec2bf-6c93-4171-9363-772e969aa333-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-jg4w7\" (UID: \"8efec2bf-6c93-4171-9363-772e969aa333\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jg4w7" Jul 18 00:47:08.643075 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.643006 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8efec2bf-6c93-4171-9363-772e969aa333-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-jg4w7\" (UID: \"8efec2bf-6c93-4171-9363-772e969aa333\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jg4w7" Jul 18 00:47:08.643075 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.643047 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8efec2bf-6c93-4171-9363-772e969aa333-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-jg4w7\" (UID: \"8efec2bf-6c93-4171-9363-772e969aa333\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jg4w7" Jul 18 00:47:08.643277 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.643078 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8efec2bf-6c93-4171-9363-772e969aa333-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-jg4w7\" (UID: \"8efec2bf-6c93-4171-9363-772e969aa333\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jg4w7" Jul 18 00:47:08.643555 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.643520 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8efec2bf-6c93-4171-9363-772e969aa333-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-jg4w7\" (UID: \"8efec2bf-6c93-4171-9363-772e969aa333\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jg4w7" Jul 18 00:47:08.645403 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.645373 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8efec2bf-6c93-4171-9363-772e969aa333-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-jg4w7\" (UID: \"8efec2bf-6c93-4171-9363-772e969aa333\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jg4w7" Jul 18 00:47:08.645527 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.645499 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8efec2bf-6c93-4171-9363-772e969aa333-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-jg4w7\" (UID: \"8efec2bf-6c93-4171-9363-772e969aa333\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jg4w7" Jul 18 00:47:08.645527 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.645518 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8efec2bf-6c93-4171-9363-772e969aa333-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-jg4w7\" (UID: \"8efec2bf-6c93-4171-9363-772e969aa333\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jg4w7" Jul 18 00:47:08.645613 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.645574 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8efec2bf-6c93-4171-9363-772e969aa333-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-jg4w7\" (UID: \"8efec2bf-6c93-4171-9363-772e969aa333\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jg4w7" Jul 18 00:47:08.651860 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.651802 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8efec2bf-6c93-4171-9363-772e969aa333-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-jg4w7\" (UID: \"8efec2bf-6c93-4171-9363-772e969aa333\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jg4w7" Jul 18 00:47:08.652015 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.651998 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hsdb\" (UniqueName: \"kubernetes.io/projected/8efec2bf-6c93-4171-9363-772e969aa333-kube-api-access-5hsdb\") pod \"istiod-openshift-gateway-94bb8fbfd-jg4w7\" (UID: \"8efec2bf-6c93-4171-9363-772e969aa333\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jg4w7" Jul 18 00:47:08.712686 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.712639 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jg4w7" Jul 18 00:47:08.888324 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:08.888299 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jg4w7"] Jul 18 00:47:08.889647 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:47:08.889624 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8efec2bf_6c93_4171_9363_772e969aa333.slice/crio-911e33a365ef43c65c90a647f3bb117b9c90acd780480a7b7b62c6ae45f9fcb5 WatchSource:0}: Error finding container 911e33a365ef43c65c90a647f3bb117b9c90acd780480a7b7b62c6ae45f9fcb5: Status 404 returned error can't find the container with id 911e33a365ef43c65c90a647f3bb117b9c90acd780480a7b7b62c6ae45f9fcb5 Jul 18 00:47:09.803840 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:09.803808 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-845f554b8f-989s7" Jul 18 00:47:09.847149 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:09.847109 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jg4w7" event={"ID":"8efec2bf-6c93-4171-9363-772e969aa333","Type":"ContainerStarted","Data":"911e33a365ef43c65c90a647f3bb117b9c90acd780480a7b7b62c6ae45f9fcb5"} Jul 18 00:47:11.020956 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:11.020897 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 18 00:47:11.021271 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:11.020989 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 18 00:47:11.858697 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:11.858638 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jg4w7" event={"ID":"8efec2bf-6c93-4171-9363-772e969aa333","Type":"ContainerStarted","Data":"896259b4add39121280f0ba8fda9668362233ae72b7c8c3d1519bf75935f5ed1"} Jul 18 00:47:11.858901 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:11.858815 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jg4w7" Jul 18 00:47:11.884811 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:11.884750 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jg4w7" podStartSLOduration=1.755803786 podStartE2EDuration="3.884733638s" totalImagesPullingTime="2.128929852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:47:11.882799511 +0000 UTC m=+392.234649857" watchObservedRunningTime="2026-07-18 00:47:11.884733638 +0000 UTC m=+392.236583994" Jul 18 00:47:12.864720 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:47:12.864661 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jg4w7" Jul 18 00:48:15.558430 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:15.558390 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv"] Jul 18 00:48:15.560731 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:15.560713 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv" Jul 18 00:48:15.563466 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:15.563438 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 18 00:48:15.564366 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:15.564350 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xm56t\"" Jul 18 00:48:15.564448 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:15.564388 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 18 00:48:15.567588 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:15.567558 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv"] Jul 18 00:48:15.705497 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:15.705456 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69a781e2-a1bd-4732-8007-717d6a3b21b6-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv\" (UID: \"69a781e2-a1bd-4732-8007-717d6a3b21b6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv" Jul 18 00:48:15.705709 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:15.705523 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzmmw\" (UniqueName: \"kubernetes.io/projected/69a781e2-a1bd-4732-8007-717d6a3b21b6-kube-api-access-tzmmw\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv\" (UID: \"69a781e2-a1bd-4732-8007-717d6a3b21b6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv" Jul 18 00:48:15.705709 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:15.705587 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69a781e2-a1bd-4732-8007-717d6a3b21b6-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv\" (UID: \"69a781e2-a1bd-4732-8007-717d6a3b21b6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv" Jul 18 00:48:15.806857 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:15.806816 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzmmw\" (UniqueName: \"kubernetes.io/projected/69a781e2-a1bd-4732-8007-717d6a3b21b6-kube-api-access-tzmmw\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv\" (UID: \"69a781e2-a1bd-4732-8007-717d6a3b21b6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv" Jul 18 00:48:15.806857 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:15.806864 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69a781e2-a1bd-4732-8007-717d6a3b21b6-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv\" (UID: \"69a781e2-a1bd-4732-8007-717d6a3b21b6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv" Jul 18 00:48:15.807101 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:15.806905 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69a781e2-a1bd-4732-8007-717d6a3b21b6-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv\" (UID: \"69a781e2-a1bd-4732-8007-717d6a3b21b6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv" Jul 18 00:48:15.807275 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:15.807256 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69a781e2-a1bd-4732-8007-717d6a3b21b6-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv\" (UID: \"69a781e2-a1bd-4732-8007-717d6a3b21b6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv" Jul 18 00:48:15.807345 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:15.807290 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69a781e2-a1bd-4732-8007-717d6a3b21b6-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv\" (UID: \"69a781e2-a1bd-4732-8007-717d6a3b21b6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv" Jul 18 00:48:15.814920 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:15.814858 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzmmw\" (UniqueName: \"kubernetes.io/projected/69a781e2-a1bd-4732-8007-717d6a3b21b6-kube-api-access-tzmmw\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv\" (UID: \"69a781e2-a1bd-4732-8007-717d6a3b21b6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv" Jul 18 00:48:15.871447 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:15.871405 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv" Jul 18 00:48:16.010353 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:16.010329 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv"] Jul 18 00:48:16.015313 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:48:16.015264 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a781e2_a1bd_4732_8007_717d6a3b21b6.slice/crio-36a4ba4c104c283e78bf3c10061e02f85e52bfe5e3779dd501cce23dd2b06c91 WatchSource:0}: Error finding container 36a4ba4c104c283e78bf3c10061e02f85e52bfe5e3779dd501cce23dd2b06c91: Status 404 returned error can't find the container with id 36a4ba4c104c283e78bf3c10061e02f85e52bfe5e3779dd501cce23dd2b06c91 Jul 18 00:48:16.136723 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:16.136690 2580 generic.go:356] "Generic (PLEG): container finished" podID="69a781e2-a1bd-4732-8007-717d6a3b21b6" containerID="4fb93b814e1fe2fe930284052b692602c884d5a15f934c0c26d2e56dbccc0f0b" exitCode=0 Jul 18 00:48:16.136898 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:16.136735 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv" event={"ID":"69a781e2-a1bd-4732-8007-717d6a3b21b6","Type":"ContainerDied","Data":"4fb93b814e1fe2fe930284052b692602c884d5a15f934c0c26d2e56dbccc0f0b"} Jul 18 00:48:16.136898 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:16.136802 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv" event={"ID":"69a781e2-a1bd-4732-8007-717d6a3b21b6","Type":"ContainerStarted","Data":"36a4ba4c104c283e78bf3c10061e02f85e52bfe5e3779dd501cce23dd2b06c91"} Jul 18 00:48:16.159230 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:16.157438 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6"] Jul 18 00:48:16.160843 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:16.160817 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6" Jul 18 00:48:16.167023 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:16.166995 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6"] Jul 18 00:48:16.311225 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:16.311177 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/388cea1d-4275-49d1-afd1-d07f2e4bedd2-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6\" (UID: \"388cea1d-4275-49d1-afd1-d07f2e4bedd2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6" Jul 18 00:48:16.311428 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:16.311319 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/388cea1d-4275-49d1-afd1-d07f2e4bedd2-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6\" (UID: \"388cea1d-4275-49d1-afd1-d07f2e4bedd2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6" Jul 18 00:48:16.311428 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:16.311357 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp8ps\" (UniqueName: \"kubernetes.io/projected/388cea1d-4275-49d1-afd1-d07f2e4bedd2-kube-api-access-hp8ps\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6\" (UID: \"388cea1d-4275-49d1-afd1-d07f2e4bedd2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6" Jul 18 00:48:16.411974 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:16.411881 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/388cea1d-4275-49d1-afd1-d07f2e4bedd2-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6\" (UID: \"388cea1d-4275-49d1-afd1-d07f2e4bedd2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6" Jul 18 00:48:16.411974 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:16.411927 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp8ps\" (UniqueName: \"kubernetes.io/projected/388cea1d-4275-49d1-afd1-d07f2e4bedd2-kube-api-access-hp8ps\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6\" (UID: \"388cea1d-4275-49d1-afd1-d07f2e4bedd2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6" Jul 18 00:48:16.411974 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:16.411972 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/388cea1d-4275-49d1-afd1-d07f2e4bedd2-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6\" (UID: \"388cea1d-4275-49d1-afd1-d07f2e4bedd2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6" Jul 18 00:48:16.412303 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:16.412272 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/388cea1d-4275-49d1-afd1-d07f2e4bedd2-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6\" (UID: \"388cea1d-4275-49d1-afd1-d07f2e4bedd2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6" Jul 18 00:48:16.412365 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:16.412321 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/388cea1d-4275-49d1-afd1-d07f2e4bedd2-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6\" (UID: \"388cea1d-4275-49d1-afd1-d07f2e4bedd2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6" Jul 18 00:48:16.420593 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:16.420561 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp8ps\" (UniqueName: \"kubernetes.io/projected/388cea1d-4275-49d1-afd1-d07f2e4bedd2-kube-api-access-hp8ps\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6\" (UID: \"388cea1d-4275-49d1-afd1-d07f2e4bedd2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6" Jul 18 00:48:16.484540 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:16.484486 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6" Jul 18 00:48:16.627736 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:16.627706 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6"] Jul 18 00:48:16.629161 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:48:16.629124 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388cea1d_4275_49d1_afd1_d07f2e4bedd2.slice/crio-565a43b9e4a9013f35076f3c307a9074430c91d7a7584ffc669d6ca7a553dd03 WatchSource:0}: Error finding container 565a43b9e4a9013f35076f3c307a9074430c91d7a7584ffc669d6ca7a553dd03: Status 404 returned error can't find the container with id 565a43b9e4a9013f35076f3c307a9074430c91d7a7584ffc669d6ca7a553dd03 Jul 18 00:48:16.755067 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:16.755028 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb"] Jul 18 00:48:16.757292 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:16.757274 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb" Jul 18 00:48:16.765207 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:16.765176 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb"] Jul 18 00:48:16.916137 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:16.916101 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b775b754-04e1-4a9e-86a8-dc0740adae1a-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb\" (UID: \"b775b754-04e1-4a9e-86a8-dc0740adae1a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb" Jul 18 00:48:16.916319 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:16.916155 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t45w\" (UniqueName: \"kubernetes.io/projected/b775b754-04e1-4a9e-86a8-dc0740adae1a-kube-api-access-7t45w\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb\" (UID: \"b775b754-04e1-4a9e-86a8-dc0740adae1a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb" Jul 18 00:48:16.916319 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:16.916297 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b775b754-04e1-4a9e-86a8-dc0740adae1a-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb\" (UID: \"b775b754-04e1-4a9e-86a8-dc0740adae1a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb" Jul 18 00:48:17.017050 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:17.017002 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b775b754-04e1-4a9e-86a8-dc0740adae1a-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb\" (UID: \"b775b754-04e1-4a9e-86a8-dc0740adae1a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb" Jul 18 00:48:17.017050 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:17.017053 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b775b754-04e1-4a9e-86a8-dc0740adae1a-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb\" (UID: \"b775b754-04e1-4a9e-86a8-dc0740adae1a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb" Jul 18 00:48:17.017323 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:17.017099 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t45w\" (UniqueName: \"kubernetes.io/projected/b775b754-04e1-4a9e-86a8-dc0740adae1a-kube-api-access-7t45w\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb\" (UID: \"b775b754-04e1-4a9e-86a8-dc0740adae1a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb" Jul 18 00:48:17.017436 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:17.017405 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b775b754-04e1-4a9e-86a8-dc0740adae1a-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb\" (UID: \"b775b754-04e1-4a9e-86a8-dc0740adae1a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb" Jul 18 00:48:17.017436 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:17.017424 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b775b754-04e1-4a9e-86a8-dc0740adae1a-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb\" (UID: \"b775b754-04e1-4a9e-86a8-dc0740adae1a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb" Jul 18 00:48:17.025892 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:17.025864 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t45w\" (UniqueName: \"kubernetes.io/projected/b775b754-04e1-4a9e-86a8-dc0740adae1a-kube-api-access-7t45w\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb\" (UID: \"b775b754-04e1-4a9e-86a8-dc0740adae1a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb" Jul 18 00:48:17.093738 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:17.093701 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb" Jul 18 00:48:17.144696 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:17.144642 2580 generic.go:356] "Generic (PLEG): container finished" podID="69a781e2-a1bd-4732-8007-717d6a3b21b6" containerID="9858a42ded7842ea1a37c42ed9cd1bf0ca7dcf735783d1b63dab95e52b589090" exitCode=0 Jul 18 00:48:17.144871 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:17.144771 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv" event={"ID":"69a781e2-a1bd-4732-8007-717d6a3b21b6","Type":"ContainerDied","Data":"9858a42ded7842ea1a37c42ed9cd1bf0ca7dcf735783d1b63dab95e52b589090"} Jul 18 00:48:17.147983 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:17.147873 2580 generic.go:356] "Generic (PLEG): container finished" podID="388cea1d-4275-49d1-afd1-d07f2e4bedd2" containerID="58729bca0e23ba168d43de2f2c088bdcf1b6dbd8a79c80fd716a6346e4cd95c0" exitCode=0 Jul 18 00:48:17.147983 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:17.147978 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6" event={"ID":"388cea1d-4275-49d1-afd1-d07f2e4bedd2","Type":"ContainerDied","Data":"58729bca0e23ba168d43de2f2c088bdcf1b6dbd8a79c80fd716a6346e4cd95c0"} Jul 18 00:48:17.148104 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:17.148024 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6" event={"ID":"388cea1d-4275-49d1-afd1-d07f2e4bedd2","Type":"ContainerStarted","Data":"565a43b9e4a9013f35076f3c307a9074430c91d7a7584ffc669d6ca7a553dd03"} Jul 18 00:48:17.241143 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:17.241104 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89"] Jul 18 00:48:17.243816 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:17.243791 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89" Jul 18 00:48:17.244966 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:17.244913 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb"] Jul 18 00:48:17.247752 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:48:17.247725 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb775b754_04e1_4a9e_86a8_dc0740adae1a.slice/crio-d4d127b4722a8cc650b618013b44664f3e6c1c50778179cc598be4e0f3d606de WatchSource:0}: Error finding container d4d127b4722a8cc650b618013b44664f3e6c1c50778179cc598be4e0f3d606de: Status 404 returned error can't find the container with id d4d127b4722a8cc650b618013b44664f3e6c1c50778179cc598be4e0f3d606de Jul 18 00:48:17.253036 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:17.252225 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89"] Jul 18 00:48:17.421117 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:17.421085 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/928dd7f6-6237-4152-adc0-0126cba4d769-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89\" (UID: \"928dd7f6-6237-4152-adc0-0126cba4d769\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89" Jul 18 00:48:17.421117 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:17.421130 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/928dd7f6-6237-4152-adc0-0126cba4d769-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89\" (UID: \"928dd7f6-6237-4152-adc0-0126cba4d769\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89" Jul 18 00:48:17.421310 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:17.421171 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h6hj\" (UniqueName: \"kubernetes.io/projected/928dd7f6-6237-4152-adc0-0126cba4d769-kube-api-access-4h6hj\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89\" (UID: \"928dd7f6-6237-4152-adc0-0126cba4d769\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89" Jul 18 00:48:17.522308 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:17.522262 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/928dd7f6-6237-4152-adc0-0126cba4d769-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89\" (UID: \"928dd7f6-6237-4152-adc0-0126cba4d769\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89" Jul 18 00:48:17.522308 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:17.522311 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/928dd7f6-6237-4152-adc0-0126cba4d769-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89\" (UID: \"928dd7f6-6237-4152-adc0-0126cba4d769\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89" Jul 18 00:48:17.522577 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:17.522357 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h6hj\" (UniqueName: \"kubernetes.io/projected/928dd7f6-6237-4152-adc0-0126cba4d769-kube-api-access-4h6hj\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89\" (UID: \"928dd7f6-6237-4152-adc0-0126cba4d769\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89" Jul 18 00:48:17.522722 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:17.522659 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/928dd7f6-6237-4152-adc0-0126cba4d769-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89\" (UID: \"928dd7f6-6237-4152-adc0-0126cba4d769\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89" Jul 18 00:48:17.522797 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:17.522756 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/928dd7f6-6237-4152-adc0-0126cba4d769-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89\" (UID: \"928dd7f6-6237-4152-adc0-0126cba4d769\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89" Jul 18 00:48:17.531261 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:17.531192 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h6hj\" (UniqueName: \"kubernetes.io/projected/928dd7f6-6237-4152-adc0-0126cba4d769-kube-api-access-4h6hj\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89\" (UID: \"928dd7f6-6237-4152-adc0-0126cba4d769\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89" Jul 18 00:48:17.556497 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:17.556457 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89" Jul 18 00:48:17.702782 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:17.702755 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89"] Jul 18 00:48:17.704783 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:48:17.704744 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod928dd7f6_6237_4152_adc0_0126cba4d769.slice/crio-12ec2bf30e5e681b62883fa7352098ff61611762219548d830a55a9775af6bdb WatchSource:0}: Error finding container 12ec2bf30e5e681b62883fa7352098ff61611762219548d830a55a9775af6bdb: Status 404 returned error can't find the container with id 12ec2bf30e5e681b62883fa7352098ff61611762219548d830a55a9775af6bdb Jul 18 00:48:18.155287 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:18.155199 2580 generic.go:356] "Generic (PLEG): container finished" podID="69a781e2-a1bd-4732-8007-717d6a3b21b6" containerID="186e72e186b77daf5582c151e29b4c069050aa9e5686cd23ac246657a7d3ee06" exitCode=0 Jul 18 00:48:18.155440 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:18.155285 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv" event={"ID":"69a781e2-a1bd-4732-8007-717d6a3b21b6","Type":"ContainerDied","Data":"186e72e186b77daf5582c151e29b4c069050aa9e5686cd23ac246657a7d3ee06"} Jul 18 00:48:18.157603 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:18.157565 2580 generic.go:356] "Generic (PLEG): container finished" podID="388cea1d-4275-49d1-afd1-d07f2e4bedd2" containerID="eca91432ad0d65a6d0ce2030620c92119c9229480531cdb105470c5121af35a1" exitCode=0 Jul 18 00:48:18.157765 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:18.157628 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6" event={"ID":"388cea1d-4275-49d1-afd1-d07f2e4bedd2","Type":"ContainerDied","Data":"eca91432ad0d65a6d0ce2030620c92119c9229480531cdb105470c5121af35a1"} Jul 18 00:48:18.159440 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:18.159403 2580 generic.go:356] "Generic (PLEG): container finished" podID="928dd7f6-6237-4152-adc0-0126cba4d769" containerID="68daaff9c0ca1665b2ed10678c0c81294b9c1b49ce59ff55f47726c78d6c4622" exitCode=0 Jul 18 00:48:18.159524 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:18.159462 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89" event={"ID":"928dd7f6-6237-4152-adc0-0126cba4d769","Type":"ContainerDied","Data":"68daaff9c0ca1665b2ed10678c0c81294b9c1b49ce59ff55f47726c78d6c4622"} Jul 18 00:48:18.159524 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:18.159510 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89" event={"ID":"928dd7f6-6237-4152-adc0-0126cba4d769","Type":"ContainerStarted","Data":"12ec2bf30e5e681b62883fa7352098ff61611762219548d830a55a9775af6bdb"} Jul 18 00:48:18.161175 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:18.161150 2580 generic.go:356] "Generic (PLEG): container finished" podID="b775b754-04e1-4a9e-86a8-dc0740adae1a" containerID="6b549ca2a5f27923c8d9a7f3cd84a4d1b8591d5c3356fe60c39abaf1fdc7f971" exitCode=0 Jul 18 00:48:18.161289 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:18.161199 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb" event={"ID":"b775b754-04e1-4a9e-86a8-dc0740adae1a","Type":"ContainerDied","Data":"6b549ca2a5f27923c8d9a7f3cd84a4d1b8591d5c3356fe60c39abaf1fdc7f971"} Jul 18 00:48:18.161289 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:18.161237 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb" event={"ID":"b775b754-04e1-4a9e-86a8-dc0740adae1a","Type":"ContainerStarted","Data":"d4d127b4722a8cc650b618013b44664f3e6c1c50778179cc598be4e0f3d606de"} Jul 18 00:48:19.168475 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:19.168389 2580 generic.go:356] "Generic (PLEG): container finished" podID="388cea1d-4275-49d1-afd1-d07f2e4bedd2" containerID="6dbd5ffac20fd85eb78f5174eb833e52afda321aee6ea96d0d8f6821a3b207f0" exitCode=0 Jul 18 00:48:19.168901 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:19.168474 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6" event={"ID":"388cea1d-4275-49d1-afd1-d07f2e4bedd2","Type":"ContainerDied","Data":"6dbd5ffac20fd85eb78f5174eb833e52afda321aee6ea96d0d8f6821a3b207f0"} Jul 18 00:48:19.170273 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:19.170251 2580 generic.go:356] "Generic (PLEG): container finished" podID="928dd7f6-6237-4152-adc0-0126cba4d769" containerID="8d40bb3f9a8cf96e0be8f7e0dcd0f12b40b28aa53f9b49c844ea6118f2afc426" exitCode=0 Jul 18 00:48:19.170385 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:19.170317 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89" event={"ID":"928dd7f6-6237-4152-adc0-0126cba4d769","Type":"ContainerDied","Data":"8d40bb3f9a8cf96e0be8f7e0dcd0f12b40b28aa53f9b49c844ea6118f2afc426"} Jul 18 00:48:19.172273 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:19.172243 2580 generic.go:356] "Generic (PLEG): container finished" podID="b775b754-04e1-4a9e-86a8-dc0740adae1a" containerID="d95d4dfeee69c3f0fe2120ff62b2eac0c82f3fc8bf458e46192519aae25f5f07" exitCode=0 Jul 18 00:48:19.172361 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:19.172347 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb" event={"ID":"b775b754-04e1-4a9e-86a8-dc0740adae1a","Type":"ContainerDied","Data":"d95d4dfeee69c3f0fe2120ff62b2eac0c82f3fc8bf458e46192519aae25f5f07"} Jul 18 00:48:19.309988 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:19.309961 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv" Jul 18 00:48:19.440081 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:19.439992 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69a781e2-a1bd-4732-8007-717d6a3b21b6-util\") pod \"69a781e2-a1bd-4732-8007-717d6a3b21b6\" (UID: \"69a781e2-a1bd-4732-8007-717d6a3b21b6\") " Jul 18 00:48:19.440081 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:19.440059 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzmmw\" (UniqueName: \"kubernetes.io/projected/69a781e2-a1bd-4732-8007-717d6a3b21b6-kube-api-access-tzmmw\") pod \"69a781e2-a1bd-4732-8007-717d6a3b21b6\" (UID: \"69a781e2-a1bd-4732-8007-717d6a3b21b6\") " Jul 18 00:48:19.440262 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:19.440100 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69a781e2-a1bd-4732-8007-717d6a3b21b6-bundle\") pod \"69a781e2-a1bd-4732-8007-717d6a3b21b6\" (UID: \"69a781e2-a1bd-4732-8007-717d6a3b21b6\") " Jul 18 00:48:19.440763 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:19.440696 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a781e2-a1bd-4732-8007-717d6a3b21b6-bundle" (OuterVolumeSpecName: "bundle") pod "69a781e2-a1bd-4732-8007-717d6a3b21b6" (UID: "69a781e2-a1bd-4732-8007-717d6a3b21b6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 00:48:19.442424 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:19.442397 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a781e2-a1bd-4732-8007-717d6a3b21b6-kube-api-access-tzmmw" (OuterVolumeSpecName: "kube-api-access-tzmmw") pod "69a781e2-a1bd-4732-8007-717d6a3b21b6" (UID: "69a781e2-a1bd-4732-8007-717d6a3b21b6"). InnerVolumeSpecName "kube-api-access-tzmmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 00:48:19.445118 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:19.445073 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a781e2-a1bd-4732-8007-717d6a3b21b6-util" (OuterVolumeSpecName: "util") pod "69a781e2-a1bd-4732-8007-717d6a3b21b6" (UID: "69a781e2-a1bd-4732-8007-717d6a3b21b6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 00:48:19.540958 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:19.540921 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69a781e2-a1bd-4732-8007-717d6a3b21b6-util\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:48:19.540958 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:19.540953 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzmmw\" (UniqueName: \"kubernetes.io/projected/69a781e2-a1bd-4732-8007-717d6a3b21b6-kube-api-access-tzmmw\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:48:19.540958 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:19.540964 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69a781e2-a1bd-4732-8007-717d6a3b21b6-bundle\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:48:20.184334 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:20.184295 2580 generic.go:356] "Generic (PLEG): container finished" podID="928dd7f6-6237-4152-adc0-0126cba4d769" containerID="f77f27f111038f704151349f007669b5cd3ba607c2b8b24997f22c033b00a3f6" exitCode=0 Jul 18 00:48:20.184737 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:20.184360 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89" event={"ID":"928dd7f6-6237-4152-adc0-0126cba4d769","Type":"ContainerDied","Data":"f77f27f111038f704151349f007669b5cd3ba607c2b8b24997f22c033b00a3f6"} Jul 18 00:48:20.186359 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:20.186331 2580 generic.go:356] "Generic (PLEG): container finished" podID="b775b754-04e1-4a9e-86a8-dc0740adae1a" containerID="1b657c736ed962169008f772404175d7f9171a61aa38a3192389d62cd3452ba9" exitCode=0 Jul 18 00:48:20.186502 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:20.186389 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb" event={"ID":"b775b754-04e1-4a9e-86a8-dc0740adae1a","Type":"ContainerDied","Data":"1b657c736ed962169008f772404175d7f9171a61aa38a3192389d62cd3452ba9"} Jul 18 00:48:20.188218 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:20.188196 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv" event={"ID":"69a781e2-a1bd-4732-8007-717d6a3b21b6","Type":"ContainerDied","Data":"36a4ba4c104c283e78bf3c10061e02f85e52bfe5e3779dd501cce23dd2b06c91"} Jul 18 00:48:20.188322 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:20.188234 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36a4ba4c104c283e78bf3c10061e02f85e52bfe5e3779dd501cce23dd2b06c91" Jul 18 00:48:20.188390 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:20.188374 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv" Jul 18 00:48:20.316370 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:20.316344 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6" Jul 18 00:48:20.347227 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:20.347198 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/388cea1d-4275-49d1-afd1-d07f2e4bedd2-bundle\") pod \"388cea1d-4275-49d1-afd1-d07f2e4bedd2\" (UID: \"388cea1d-4275-49d1-afd1-d07f2e4bedd2\") " Jul 18 00:48:20.347396 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:20.347257 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hp8ps\" (UniqueName: \"kubernetes.io/projected/388cea1d-4275-49d1-afd1-d07f2e4bedd2-kube-api-access-hp8ps\") pod \"388cea1d-4275-49d1-afd1-d07f2e4bedd2\" (UID: \"388cea1d-4275-49d1-afd1-d07f2e4bedd2\") " Jul 18 00:48:20.347396 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:20.347323 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/388cea1d-4275-49d1-afd1-d07f2e4bedd2-util\") pod \"388cea1d-4275-49d1-afd1-d07f2e4bedd2\" (UID: \"388cea1d-4275-49d1-afd1-d07f2e4bedd2\") " Jul 18 00:48:20.347804 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:20.347772 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388cea1d-4275-49d1-afd1-d07f2e4bedd2-bundle" (OuterVolumeSpecName: "bundle") pod "388cea1d-4275-49d1-afd1-d07f2e4bedd2" (UID: "388cea1d-4275-49d1-afd1-d07f2e4bedd2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 00:48:20.349546 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:20.349521 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388cea1d-4275-49d1-afd1-d07f2e4bedd2-kube-api-access-hp8ps" (OuterVolumeSpecName: "kube-api-access-hp8ps") pod "388cea1d-4275-49d1-afd1-d07f2e4bedd2" (UID: "388cea1d-4275-49d1-afd1-d07f2e4bedd2"). InnerVolumeSpecName "kube-api-access-hp8ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 00:48:20.353563 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:20.353524 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388cea1d-4275-49d1-afd1-d07f2e4bedd2-util" (OuterVolumeSpecName: "util") pod "388cea1d-4275-49d1-afd1-d07f2e4bedd2" (UID: "388cea1d-4275-49d1-afd1-d07f2e4bedd2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 00:48:20.448265 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:20.448155 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/388cea1d-4275-49d1-afd1-d07f2e4bedd2-util\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:48:20.448265 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:20.448189 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/388cea1d-4275-49d1-afd1-d07f2e4bedd2-bundle\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:48:20.448265 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:20.448203 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp8ps\" (UniqueName: \"kubernetes.io/projected/388cea1d-4275-49d1-afd1-d07f2e4bedd2-kube-api-access-hp8ps\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:48:21.195220 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:21.195189 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6" Jul 18 00:48:21.195220 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:21.195212 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6" event={"ID":"388cea1d-4275-49d1-afd1-d07f2e4bedd2","Type":"ContainerDied","Data":"565a43b9e4a9013f35076f3c307a9074430c91d7a7584ffc669d6ca7a553dd03"} Jul 18 00:48:21.195731 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:21.195255 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="565a43b9e4a9013f35076f3c307a9074430c91d7a7584ffc669d6ca7a553dd03" Jul 18 00:48:21.324426 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:21.324403 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb" Jul 18 00:48:21.353948 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:21.353918 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89" Jul 18 00:48:21.356834 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:21.356807 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7t45w\" (UniqueName: \"kubernetes.io/projected/b775b754-04e1-4a9e-86a8-dc0740adae1a-kube-api-access-7t45w\") pod \"b775b754-04e1-4a9e-86a8-dc0740adae1a\" (UID: \"b775b754-04e1-4a9e-86a8-dc0740adae1a\") " Jul 18 00:48:21.356978 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:21.356856 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b775b754-04e1-4a9e-86a8-dc0740adae1a-bundle\") pod \"b775b754-04e1-4a9e-86a8-dc0740adae1a\" (UID: \"b775b754-04e1-4a9e-86a8-dc0740adae1a\") " Jul 18 00:48:21.356978 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:21.356928 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b775b754-04e1-4a9e-86a8-dc0740adae1a-util\") pod \"b775b754-04e1-4a9e-86a8-dc0740adae1a\" (UID: \"b775b754-04e1-4a9e-86a8-dc0740adae1a\") " Jul 18 00:48:21.357778 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:21.357596 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b775b754-04e1-4a9e-86a8-dc0740adae1a-bundle" (OuterVolumeSpecName: "bundle") pod "b775b754-04e1-4a9e-86a8-dc0740adae1a" (UID: "b775b754-04e1-4a9e-86a8-dc0740adae1a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 00:48:21.357952 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:21.357931 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b775b754-04e1-4a9e-86a8-dc0740adae1a-bundle\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:48:21.364537 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:21.364505 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b775b754-04e1-4a9e-86a8-dc0740adae1a-kube-api-access-7t45w" (OuterVolumeSpecName: "kube-api-access-7t45w") pod "b775b754-04e1-4a9e-86a8-dc0740adae1a" (UID: "b775b754-04e1-4a9e-86a8-dc0740adae1a"). InnerVolumeSpecName "kube-api-access-7t45w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 00:48:21.366685 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:21.366639 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b775b754-04e1-4a9e-86a8-dc0740adae1a-util" (OuterVolumeSpecName: "util") pod "b775b754-04e1-4a9e-86a8-dc0740adae1a" (UID: "b775b754-04e1-4a9e-86a8-dc0740adae1a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 00:48:21.459013 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:21.458851 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4h6hj\" (UniqueName: \"kubernetes.io/projected/928dd7f6-6237-4152-adc0-0126cba4d769-kube-api-access-4h6hj\") pod \"928dd7f6-6237-4152-adc0-0126cba4d769\" (UID: \"928dd7f6-6237-4152-adc0-0126cba4d769\") " Jul 18 00:48:21.459013 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:21.458915 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/928dd7f6-6237-4152-adc0-0126cba4d769-util\") pod \"928dd7f6-6237-4152-adc0-0126cba4d769\" (UID: \"928dd7f6-6237-4152-adc0-0126cba4d769\") " Jul 18 00:48:21.459240 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:21.459020 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/928dd7f6-6237-4152-adc0-0126cba4d769-bundle\") pod \"928dd7f6-6237-4152-adc0-0126cba4d769\" (UID: \"928dd7f6-6237-4152-adc0-0126cba4d769\") " Jul 18 00:48:21.459301 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:21.459288 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b775b754-04e1-4a9e-86a8-dc0740adae1a-util\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:48:21.459364 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:21.459307 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t45w\" (UniqueName: \"kubernetes.io/projected/b775b754-04e1-4a9e-86a8-dc0740adae1a-kube-api-access-7t45w\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:48:21.459890 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:21.459860 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/928dd7f6-6237-4152-adc0-0126cba4d769-bundle" (OuterVolumeSpecName: "bundle") pod "928dd7f6-6237-4152-adc0-0126cba4d769" (UID: "928dd7f6-6237-4152-adc0-0126cba4d769"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 00:48:21.460961 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:21.460926 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/928dd7f6-6237-4152-adc0-0126cba4d769-kube-api-access-4h6hj" (OuterVolumeSpecName: "kube-api-access-4h6hj") pod "928dd7f6-6237-4152-adc0-0126cba4d769" (UID: "928dd7f6-6237-4152-adc0-0126cba4d769"). InnerVolumeSpecName "kube-api-access-4h6hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 00:48:21.464830 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:21.464803 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/928dd7f6-6237-4152-adc0-0126cba4d769-util" (OuterVolumeSpecName: "util") pod "928dd7f6-6237-4152-adc0-0126cba4d769" (UID: "928dd7f6-6237-4152-adc0-0126cba4d769"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 00:48:21.560579 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:21.560537 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h6hj\" (UniqueName: \"kubernetes.io/projected/928dd7f6-6237-4152-adc0-0126cba4d769-kube-api-access-4h6hj\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:48:21.560579 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:21.560573 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/928dd7f6-6237-4152-adc0-0126cba4d769-util\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:48:21.560579 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:21.560583 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/928dd7f6-6237-4152-adc0-0126cba4d769-bundle\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:48:22.202408 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:22.202374 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89" event={"ID":"928dd7f6-6237-4152-adc0-0126cba4d769","Type":"ContainerDied","Data":"12ec2bf30e5e681b62883fa7352098ff61611762219548d830a55a9775af6bdb"} Jul 18 00:48:22.202841 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:22.202437 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12ec2bf30e5e681b62883fa7352098ff61611762219548d830a55a9775af6bdb" Jul 18 00:48:22.202841 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:22.202387 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89" Jul 18 00:48:22.204596 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:22.204567 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb" event={"ID":"b775b754-04e1-4a9e-86a8-dc0740adae1a","Type":"ContainerDied","Data":"d4d127b4722a8cc650b618013b44664f3e6c1c50778179cc598be4e0f3d606de"} Jul 18 00:48:22.204596 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:22.204588 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb" Jul 18 00:48:22.204795 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:22.204613 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4d127b4722a8cc650b618013b44664f3e6c1c50778179cc598be4e0f3d606de" Jul 18 00:48:41.833285 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.833251 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b7b88f5d8-ttdwm"] Jul 18 00:48:41.833682 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.833583 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69a781e2-a1bd-4732-8007-717d6a3b21b6" containerName="extract" Jul 18 00:48:41.833682 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.833596 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="69a781e2-a1bd-4732-8007-717d6a3b21b6" containerName="extract" Jul 18 00:48:41.833682 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.833604 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="928dd7f6-6237-4152-adc0-0126cba4d769" containerName="pull" Jul 18 00:48:41.833682 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.833610 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="928dd7f6-6237-4152-adc0-0126cba4d769" containerName="pull" Jul 18 00:48:41.833682 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.833622 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="928dd7f6-6237-4152-adc0-0126cba4d769" containerName="extract" Jul 18 00:48:41.833682 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.833631 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="928dd7f6-6237-4152-adc0-0126cba4d769" containerName="extract" Jul 18 00:48:41.833682 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.833643 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b775b754-04e1-4a9e-86a8-dc0740adae1a" containerName="pull" Jul 18 00:48:41.833682 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.833649 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b775b754-04e1-4a9e-86a8-dc0740adae1a" containerName="pull" Jul 18 00:48:41.833682 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.833662 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="388cea1d-4275-49d1-afd1-d07f2e4bedd2" containerName="util" Jul 18 00:48:41.833682 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.833682 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="388cea1d-4275-49d1-afd1-d07f2e4bedd2" containerName="util" Jul 18 00:48:41.833682 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.833688 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="388cea1d-4275-49d1-afd1-d07f2e4bedd2" containerName="extract" Jul 18 00:48:41.834012 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.833693 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="388cea1d-4275-49d1-afd1-d07f2e4bedd2" containerName="extract" Jul 18 00:48:41.834012 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.833700 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69a781e2-a1bd-4732-8007-717d6a3b21b6" containerName="util" Jul 18 00:48:41.834012 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.833711 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="69a781e2-a1bd-4732-8007-717d6a3b21b6" containerName="util" Jul 18 00:48:41.834012 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.833721 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="388cea1d-4275-49d1-afd1-d07f2e4bedd2" containerName="pull" Jul 18 00:48:41.834012 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.833726 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="388cea1d-4275-49d1-afd1-d07f2e4bedd2" containerName="pull" Jul 18 00:48:41.834012 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.833731 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b775b754-04e1-4a9e-86a8-dc0740adae1a" containerName="extract" Jul 18 00:48:41.834012 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.833736 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b775b754-04e1-4a9e-86a8-dc0740adae1a" containerName="extract" Jul 18 00:48:41.834012 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.833742 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="928dd7f6-6237-4152-adc0-0126cba4d769" containerName="util" Jul 18 00:48:41.834012 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.833746 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="928dd7f6-6237-4152-adc0-0126cba4d769" containerName="util" Jul 18 00:48:41.834012 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.833753 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69a781e2-a1bd-4732-8007-717d6a3b21b6" containerName="pull" Jul 18 00:48:41.834012 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.833757 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="69a781e2-a1bd-4732-8007-717d6a3b21b6" containerName="pull" Jul 18 00:48:41.834012 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.833763 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b775b754-04e1-4a9e-86a8-dc0740adae1a" containerName="util" Jul 18 00:48:41.834012 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.833767 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b775b754-04e1-4a9e-86a8-dc0740adae1a" containerName="util" Jul 18 00:48:41.834012 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.833820 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="69a781e2-a1bd-4732-8007-717d6a3b21b6" containerName="extract" Jul 18 00:48:41.834012 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.833828 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="388cea1d-4275-49d1-afd1-d07f2e4bedd2" containerName="extract" Jul 18 00:48:41.834012 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.833835 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="928dd7f6-6237-4152-adc0-0126cba4d769" containerName="extract" Jul 18 00:48:41.834012 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.833840 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b775b754-04e1-4a9e-86a8-dc0740adae1a" containerName="extract" Jul 18 00:48:41.835646 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.835629 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b7b88f5d8-ttdwm" Jul 18 00:48:41.854331 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.854298 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b7b88f5d8-ttdwm"] Jul 18 00:48:41.929258 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.929214 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddm4d\" (UniqueName: \"kubernetes.io/projected/17efbe25-7468-46c3-9df0-cbdf2611bbef-kube-api-access-ddm4d\") pod \"console-5b7b88f5d8-ttdwm\" (UID: \"17efbe25-7468-46c3-9df0-cbdf2611bbef\") " pod="openshift-console/console-5b7b88f5d8-ttdwm" Jul 18 00:48:41.929258 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.929257 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17efbe25-7468-46c3-9df0-cbdf2611bbef-oauth-serving-cert\") pod \"console-5b7b88f5d8-ttdwm\" (UID: \"17efbe25-7468-46c3-9df0-cbdf2611bbef\") " pod="openshift-console/console-5b7b88f5d8-ttdwm" Jul 18 00:48:41.929471 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.929277 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17efbe25-7468-46c3-9df0-cbdf2611bbef-console-config\") pod \"console-5b7b88f5d8-ttdwm\" (UID: \"17efbe25-7468-46c3-9df0-cbdf2611bbef\") " pod="openshift-console/console-5b7b88f5d8-ttdwm" Jul 18 00:48:41.929471 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.929296 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17efbe25-7468-46c3-9df0-cbdf2611bbef-console-oauth-config\") pod \"console-5b7b88f5d8-ttdwm\" (UID: \"17efbe25-7468-46c3-9df0-cbdf2611bbef\") " pod="openshift-console/console-5b7b88f5d8-ttdwm" Jul 18 00:48:41.929471 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.929361 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17efbe25-7468-46c3-9df0-cbdf2611bbef-service-ca\") pod \"console-5b7b88f5d8-ttdwm\" (UID: \"17efbe25-7468-46c3-9df0-cbdf2611bbef\") " pod="openshift-console/console-5b7b88f5d8-ttdwm" Jul 18 00:48:41.929471 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.929414 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17efbe25-7468-46c3-9df0-cbdf2611bbef-trusted-ca-bundle\") pod \"console-5b7b88f5d8-ttdwm\" (UID: \"17efbe25-7468-46c3-9df0-cbdf2611bbef\") " pod="openshift-console/console-5b7b88f5d8-ttdwm" Jul 18 00:48:41.929471 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:41.929462 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17efbe25-7468-46c3-9df0-cbdf2611bbef-console-serving-cert\") pod \"console-5b7b88f5d8-ttdwm\" (UID: \"17efbe25-7468-46c3-9df0-cbdf2611bbef\") " pod="openshift-console/console-5b7b88f5d8-ttdwm" Jul 18 00:48:42.030981 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:42.030938 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddm4d\" (UniqueName: \"kubernetes.io/projected/17efbe25-7468-46c3-9df0-cbdf2611bbef-kube-api-access-ddm4d\") pod \"console-5b7b88f5d8-ttdwm\" (UID: \"17efbe25-7468-46c3-9df0-cbdf2611bbef\") " pod="openshift-console/console-5b7b88f5d8-ttdwm" Jul 18 00:48:42.031121 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:42.030994 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17efbe25-7468-46c3-9df0-cbdf2611bbef-oauth-serving-cert\") pod \"console-5b7b88f5d8-ttdwm\" (UID: \"17efbe25-7468-46c3-9df0-cbdf2611bbef\") " pod="openshift-console/console-5b7b88f5d8-ttdwm" Jul 18 00:48:42.031121 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:42.031025 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17efbe25-7468-46c3-9df0-cbdf2611bbef-console-config\") pod \"console-5b7b88f5d8-ttdwm\" (UID: \"17efbe25-7468-46c3-9df0-cbdf2611bbef\") " pod="openshift-console/console-5b7b88f5d8-ttdwm" Jul 18 00:48:42.031121 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:42.031055 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17efbe25-7468-46c3-9df0-cbdf2611bbef-console-oauth-config\") pod \"console-5b7b88f5d8-ttdwm\" (UID: \"17efbe25-7468-46c3-9df0-cbdf2611bbef\") " pod="openshift-console/console-5b7b88f5d8-ttdwm" Jul 18 00:48:42.031121 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:42.031105 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17efbe25-7468-46c3-9df0-cbdf2611bbef-service-ca\") pod \"console-5b7b88f5d8-ttdwm\" (UID: \"17efbe25-7468-46c3-9df0-cbdf2611bbef\") " pod="openshift-console/console-5b7b88f5d8-ttdwm" Jul 18 00:48:42.031336 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:42.031163 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17efbe25-7468-46c3-9df0-cbdf2611bbef-trusted-ca-bundle\") pod \"console-5b7b88f5d8-ttdwm\" (UID: \"17efbe25-7468-46c3-9df0-cbdf2611bbef\") " pod="openshift-console/console-5b7b88f5d8-ttdwm" Jul 18 00:48:42.031336 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:42.031240 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17efbe25-7468-46c3-9df0-cbdf2611bbef-console-serving-cert\") pod \"console-5b7b88f5d8-ttdwm\" (UID: \"17efbe25-7468-46c3-9df0-cbdf2611bbef\") " pod="openshift-console/console-5b7b88f5d8-ttdwm" Jul 18 00:48:42.031894 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:42.031867 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17efbe25-7468-46c3-9df0-cbdf2611bbef-console-config\") pod \"console-5b7b88f5d8-ttdwm\" (UID: \"17efbe25-7468-46c3-9df0-cbdf2611bbef\") " pod="openshift-console/console-5b7b88f5d8-ttdwm" Jul 18 00:48:42.032013 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:42.031867 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17efbe25-7468-46c3-9df0-cbdf2611bbef-oauth-serving-cert\") pod \"console-5b7b88f5d8-ttdwm\" (UID: \"17efbe25-7468-46c3-9df0-cbdf2611bbef\") " pod="openshift-console/console-5b7b88f5d8-ttdwm" Jul 18 00:48:42.032013 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:42.031967 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17efbe25-7468-46c3-9df0-cbdf2611bbef-service-ca\") pod \"console-5b7b88f5d8-ttdwm\" (UID: \"17efbe25-7468-46c3-9df0-cbdf2611bbef\") " pod="openshift-console/console-5b7b88f5d8-ttdwm" Jul 18 00:48:42.032138 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:42.032055 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17efbe25-7468-46c3-9df0-cbdf2611bbef-trusted-ca-bundle\") pod \"console-5b7b88f5d8-ttdwm\" (UID: \"17efbe25-7468-46c3-9df0-cbdf2611bbef\") " pod="openshift-console/console-5b7b88f5d8-ttdwm" Jul 18 00:48:42.034208 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:42.034183 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17efbe25-7468-46c3-9df0-cbdf2611bbef-console-oauth-config\") pod \"console-5b7b88f5d8-ttdwm\" (UID: \"17efbe25-7468-46c3-9df0-cbdf2611bbef\") " pod="openshift-console/console-5b7b88f5d8-ttdwm" Jul 18 00:48:42.034389 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:42.034367 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17efbe25-7468-46c3-9df0-cbdf2611bbef-console-serving-cert\") pod \"console-5b7b88f5d8-ttdwm\" (UID: \"17efbe25-7468-46c3-9df0-cbdf2611bbef\") " pod="openshift-console/console-5b7b88f5d8-ttdwm" Jul 18 00:48:42.038654 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:42.038630 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddm4d\" (UniqueName: \"kubernetes.io/projected/17efbe25-7468-46c3-9df0-cbdf2611bbef-kube-api-access-ddm4d\") pod \"console-5b7b88f5d8-ttdwm\" (UID: \"17efbe25-7468-46c3-9df0-cbdf2611bbef\") " pod="openshift-console/console-5b7b88f5d8-ttdwm" Jul 18 00:48:42.144924 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:42.144828 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b7b88f5d8-ttdwm" Jul 18 00:48:42.284711 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:42.284662 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b7b88f5d8-ttdwm"] Jul 18 00:48:42.286285 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:48:42.286245 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17efbe25_7468_46c3_9df0_cbdf2611bbef.slice/crio-14ba18974b60e70d3d6a097ed1b97f74289782287928d9a46fade30757d68c6e WatchSource:0}: Error finding container 14ba18974b60e70d3d6a097ed1b97f74289782287928d9a46fade30757d68c6e: Status 404 returned error can't find the container with id 14ba18974b60e70d3d6a097ed1b97f74289782287928d9a46fade30757d68c6e Jul 18 00:48:42.299477 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:42.299444 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b7b88f5d8-ttdwm" event={"ID":"17efbe25-7468-46c3-9df0-cbdf2611bbef","Type":"ContainerStarted","Data":"14ba18974b60e70d3d6a097ed1b97f74289782287928d9a46fade30757d68c6e"} Jul 18 00:48:43.306751 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:43.306701 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b7b88f5d8-ttdwm" event={"ID":"17efbe25-7468-46c3-9df0-cbdf2611bbef","Type":"ContainerStarted","Data":"b95e047bb2e68b487f297504f9ce3abb281d02f7a2b5a4787cda8f250d0e534f"} Jul 18 00:48:43.340141 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:43.340087 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b7b88f5d8-ttdwm" podStartSLOduration=2.340069702 podStartE2EDuration="2.340069702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:48:43.337458561 +0000 UTC m=+483.689308906" watchObservedRunningTime="2026-07-18 00:48:43.340069702 +0000 UTC m=+483.691920046" Jul 18 00:48:52.149226 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:52.149197 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b7b88f5d8-ttdwm" Jul 18 00:48:52.149827 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:52.149247 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b7b88f5d8-ttdwm" Jul 18 00:48:52.150877 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:52.150860 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b7b88f5d8-ttdwm" Jul 18 00:48:52.359645 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:52.359617 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b7b88f5d8-ttdwm" Jul 18 00:48:52.452414 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:48:52.452325 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65dd5fc6d-9rctc"] Jul 18 00:49:17.473445 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:17.473384 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-65dd5fc6d-9rctc" podUID="f87668bd-29be-47d3-a838-44177f22d563" containerName="console" containerID="cri-o://a3517d50e1ec6551fc01d904261694721b1c299358b3a2359c4fd53d4d39ae34" gracePeriod=15 Jul 18 00:49:17.723803 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:17.723743 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65dd5fc6d-9rctc" Jul 18 00:49:17.757262 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:17.757233 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f87668bd-29be-47d3-a838-44177f22d563-console-serving-cert\") pod \"f87668bd-29be-47d3-a838-44177f22d563\" (UID: \"f87668bd-29be-47d3-a838-44177f22d563\") " Jul 18 00:49:17.757262 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:17.757265 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snkmz\" (UniqueName: \"kubernetes.io/projected/f87668bd-29be-47d3-a838-44177f22d563-kube-api-access-snkmz\") pod \"f87668bd-29be-47d3-a838-44177f22d563\" (UID: \"f87668bd-29be-47d3-a838-44177f22d563\") " Jul 18 00:49:17.757526 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:17.757342 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f87668bd-29be-47d3-a838-44177f22d563-oauth-serving-cert\") pod \"f87668bd-29be-47d3-a838-44177f22d563\" (UID: \"f87668bd-29be-47d3-a838-44177f22d563\") " Jul 18 00:49:17.757526 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:17.757391 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f87668bd-29be-47d3-a838-44177f22d563-console-config\") pod \"f87668bd-29be-47d3-a838-44177f22d563\" (UID: \"f87668bd-29be-47d3-a838-44177f22d563\") " Jul 18 00:49:17.757526 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:17.757412 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f87668bd-29be-47d3-a838-44177f22d563-service-ca\") pod \"f87668bd-29be-47d3-a838-44177f22d563\" (UID: \"f87668bd-29be-47d3-a838-44177f22d563\") " Jul 18 00:49:17.757526 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:17.757434 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f87668bd-29be-47d3-a838-44177f22d563-console-oauth-config\") pod \"f87668bd-29be-47d3-a838-44177f22d563\" (UID: \"f87668bd-29be-47d3-a838-44177f22d563\") " Jul 18 00:49:17.757526 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:17.757489 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f87668bd-29be-47d3-a838-44177f22d563-trusted-ca-bundle\") pod \"f87668bd-29be-47d3-a838-44177f22d563\" (UID: \"f87668bd-29be-47d3-a838-44177f22d563\") " Jul 18 00:49:17.757934 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:17.757850 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f87668bd-29be-47d3-a838-44177f22d563-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f87668bd-29be-47d3-a838-44177f22d563" (UID: "f87668bd-29be-47d3-a838-44177f22d563"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 00:49:17.758060 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:17.757928 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f87668bd-29be-47d3-a838-44177f22d563-console-config" (OuterVolumeSpecName: "console-config") pod "f87668bd-29be-47d3-a838-44177f22d563" (UID: "f87668bd-29be-47d3-a838-44177f22d563"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 00:49:17.758060 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:17.757967 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f87668bd-29be-47d3-a838-44177f22d563-service-ca" (OuterVolumeSpecName: "service-ca") pod "f87668bd-29be-47d3-a838-44177f22d563" (UID: "f87668bd-29be-47d3-a838-44177f22d563"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 00:49:17.758060 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:17.758014 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f87668bd-29be-47d3-a838-44177f22d563-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f87668bd-29be-47d3-a838-44177f22d563" (UID: "f87668bd-29be-47d3-a838-44177f22d563"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 00:49:17.759851 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:17.759825 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f87668bd-29be-47d3-a838-44177f22d563-kube-api-access-snkmz" (OuterVolumeSpecName: "kube-api-access-snkmz") pod "f87668bd-29be-47d3-a838-44177f22d563" (UID: "f87668bd-29be-47d3-a838-44177f22d563"). InnerVolumeSpecName "kube-api-access-snkmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 00:49:17.760089 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:17.760046 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f87668bd-29be-47d3-a838-44177f22d563-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f87668bd-29be-47d3-a838-44177f22d563" (UID: "f87668bd-29be-47d3-a838-44177f22d563"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 00:49:17.760197 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:17.760141 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f87668bd-29be-47d3-a838-44177f22d563-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f87668bd-29be-47d3-a838-44177f22d563" (UID: "f87668bd-29be-47d3-a838-44177f22d563"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 00:49:17.858562 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:17.858523 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f87668bd-29be-47d3-a838-44177f22d563-console-config\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:49:17.858562 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:17.858553 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f87668bd-29be-47d3-a838-44177f22d563-service-ca\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:49:17.858562 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:17.858564 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f87668bd-29be-47d3-a838-44177f22d563-console-oauth-config\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:49:17.858562 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:17.858572 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f87668bd-29be-47d3-a838-44177f22d563-trusted-ca-bundle\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:49:17.858963 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:17.858580 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f87668bd-29be-47d3-a838-44177f22d563-console-serving-cert\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:49:17.858963 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:17.858589 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snkmz\" (UniqueName: \"kubernetes.io/projected/f87668bd-29be-47d3-a838-44177f22d563-kube-api-access-snkmz\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:49:17.858963 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:17.858597 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f87668bd-29be-47d3-a838-44177f22d563-oauth-serving-cert\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:49:18.479975 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:18.479941 2580 generic.go:356] "Generic (PLEG): container finished" podID="f87668bd-29be-47d3-a838-44177f22d563" containerID="a3517d50e1ec6551fc01d904261694721b1c299358b3a2359c4fd53d4d39ae34" exitCode=0 Jul 18 00:49:18.480374 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:18.479993 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65dd5fc6d-9rctc" Jul 18 00:49:18.480374 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:18.480037 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65dd5fc6d-9rctc" event={"ID":"f87668bd-29be-47d3-a838-44177f22d563","Type":"ContainerDied","Data":"a3517d50e1ec6551fc01d904261694721b1c299358b3a2359c4fd53d4d39ae34"} Jul 18 00:49:18.480374 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:18.480090 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65dd5fc6d-9rctc" event={"ID":"f87668bd-29be-47d3-a838-44177f22d563","Type":"ContainerDied","Data":"d789947aba873d67ef0ee9a5fb5c7be19d59303423951a42cbe522b5120b003b"} Jul 18 00:49:18.480374 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:18.480111 2580 scope.go:117] "RemoveContainer" containerID="a3517d50e1ec6551fc01d904261694721b1c299358b3a2359c4fd53d4d39ae34" Jul 18 00:49:18.489056 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:18.489037 2580 scope.go:117] "RemoveContainer" containerID="a3517d50e1ec6551fc01d904261694721b1c299358b3a2359c4fd53d4d39ae34" Jul 18 00:49:18.489320 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:49:18.489300 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3517d50e1ec6551fc01d904261694721b1c299358b3a2359c4fd53d4d39ae34\": container with ID starting with a3517d50e1ec6551fc01d904261694721b1c299358b3a2359c4fd53d4d39ae34 not found: ID does not exist" containerID="a3517d50e1ec6551fc01d904261694721b1c299358b3a2359c4fd53d4d39ae34" Jul 18 00:49:18.489368 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:18.489329 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3517d50e1ec6551fc01d904261694721b1c299358b3a2359c4fd53d4d39ae34"} err="failed to get container status \"a3517d50e1ec6551fc01d904261694721b1c299358b3a2359c4fd53d4d39ae34\": rpc error: code = NotFound desc = could not find container \"a3517d50e1ec6551fc01d904261694721b1c299358b3a2359c4fd53d4d39ae34\": container with ID starting with a3517d50e1ec6551fc01d904261694721b1c299358b3a2359c4fd53d4d39ae34 not found: ID does not exist" Jul 18 00:49:18.497758 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:18.497725 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65dd5fc6d-9rctc"] Jul 18 00:49:18.507624 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:18.507599 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65dd5fc6d-9rctc"] Jul 18 00:49:20.149851 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:20.149822 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f87668bd-29be-47d3-a838-44177f22d563" path="/var/lib/kubelet/pods/f87668bd-29be-47d3-a838-44177f22d563/volumes" Jul 18 00:49:36.646663 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:36.646576 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-j5tvk"] Jul 18 00:49:36.647173 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:36.646881 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f87668bd-29be-47d3-a838-44177f22d563" containerName="console" Jul 18 00:49:36.647173 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:36.646893 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f87668bd-29be-47d3-a838-44177f22d563" containerName="console" Jul 18 00:49:36.647173 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:36.646953 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f87668bd-29be-47d3-a838-44177f22d563" containerName="console" Jul 18 00:49:36.649783 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:36.649766 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-j5tvk" Jul 18 00:49:36.652233 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:36.652212 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 18 00:49:36.652498 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:36.652480 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 18 00:49:36.653277 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:36.653263 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 18 00:49:36.653341 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:36.653265 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xm56t\"" Jul 18 00:49:36.660731 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:36.660708 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-j5tvk"] Jul 18 00:49:36.719359 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:36.719325 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzmm7\" (UniqueName: \"kubernetes.io/projected/b1236780-03ae-4b01-8493-62c691216f9c-kube-api-access-jzmm7\") pod \"limitador-limitador-64c94fd64f-j5tvk\" (UID: \"b1236780-03ae-4b01-8493-62c691216f9c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-j5tvk" Jul 18 00:49:36.719506 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:36.719379 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b1236780-03ae-4b01-8493-62c691216f9c-config-file\") pod \"limitador-limitador-64c94fd64f-j5tvk\" (UID: \"b1236780-03ae-4b01-8493-62c691216f9c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-j5tvk" Jul 18 00:49:36.756284 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:36.756254 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-j5tvk"] Jul 18 00:49:36.820391 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:36.820351 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzmm7\" (UniqueName: \"kubernetes.io/projected/b1236780-03ae-4b01-8493-62c691216f9c-kube-api-access-jzmm7\") pod \"limitador-limitador-64c94fd64f-j5tvk\" (UID: \"b1236780-03ae-4b01-8493-62c691216f9c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-j5tvk" Jul 18 00:49:36.820391 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:36.820392 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b1236780-03ae-4b01-8493-62c691216f9c-config-file\") pod \"limitador-limitador-64c94fd64f-j5tvk\" (UID: \"b1236780-03ae-4b01-8493-62c691216f9c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-j5tvk" Jul 18 00:49:36.821098 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:36.821079 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b1236780-03ae-4b01-8493-62c691216f9c-config-file\") pod \"limitador-limitador-64c94fd64f-j5tvk\" (UID: \"b1236780-03ae-4b01-8493-62c691216f9c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-j5tvk" Jul 18 00:49:36.831026 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:36.830992 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzmm7\" (UniqueName: \"kubernetes.io/projected/b1236780-03ae-4b01-8493-62c691216f9c-kube-api-access-jzmm7\") pod \"limitador-limitador-64c94fd64f-j5tvk\" (UID: \"b1236780-03ae-4b01-8493-62c691216f9c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-j5tvk" Jul 18 00:49:36.960721 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:36.960604 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-j5tvk" Jul 18 00:49:37.094575 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:37.094549 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-j5tvk"] Jul 18 00:49:37.096289 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:49:37.096251 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1236780_03ae_4b01_8493_62c691216f9c.slice/crio-4cab5f37dd6c570c660c4e867b3444a887e0b6c40cbd60baf86b9a5b37e8f43e WatchSource:0}: Error finding container 4cab5f37dd6c570c660c4e867b3444a887e0b6c40cbd60baf86b9a5b37e8f43e: Status 404 returned error can't find the container with id 4cab5f37dd6c570c660c4e867b3444a887e0b6c40cbd60baf86b9a5b37e8f43e Jul 18 00:49:37.570544 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:37.570507 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-j5tvk" event={"ID":"b1236780-03ae-4b01-8493-62c691216f9c","Type":"ContainerStarted","Data":"4cab5f37dd6c570c660c4e867b3444a887e0b6c40cbd60baf86b9a5b37e8f43e"} Jul 18 00:49:40.588472 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:40.588436 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-j5tvk" event={"ID":"b1236780-03ae-4b01-8493-62c691216f9c","Type":"ContainerStarted","Data":"f1e4281dc0c819d669464eef3c4ecadd61434986c730398f7e004bd9c95b29da"} Jul 18 00:49:40.588908 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:40.588690 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-j5tvk" Jul 18 00:49:40.603861 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:40.603815 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-j5tvk" podStartSLOduration=1.840872643 podStartE2EDuration="4.603800557s" totalImagesPullingTime="2.762927914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:49:40.60250248 +0000 UTC m=+540.954352827" watchObservedRunningTime="2026-07-18 00:49:40.603800557 +0000 UTC m=+540.955650906" Jul 18 00:49:51.594814 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:51.594783 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-j5tvk" Jul 18 00:49:51.709870 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:51.709832 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-j5tvk"] Jul 18 00:49:51.710062 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:51.710027 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-j5tvk" podUID="b1236780-03ae-4b01-8493-62c691216f9c" containerName="limitador" containerID="cri-o://f1e4281dc0c819d669464eef3c4ecadd61434986c730398f7e004bd9c95b29da" gracePeriod=30 Jul 18 00:49:52.256920 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:52.256893 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-j5tvk" Jul 18 00:49:52.359898 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:52.359819 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzmm7\" (UniqueName: \"kubernetes.io/projected/b1236780-03ae-4b01-8493-62c691216f9c-kube-api-access-jzmm7\") pod \"b1236780-03ae-4b01-8493-62c691216f9c\" (UID: \"b1236780-03ae-4b01-8493-62c691216f9c\") " Jul 18 00:49:52.359898 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:52.359872 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b1236780-03ae-4b01-8493-62c691216f9c-config-file\") pod \"b1236780-03ae-4b01-8493-62c691216f9c\" (UID: \"b1236780-03ae-4b01-8493-62c691216f9c\") " Jul 18 00:49:52.360250 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:52.360224 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1236780-03ae-4b01-8493-62c691216f9c-config-file" (OuterVolumeSpecName: "config-file") pod "b1236780-03ae-4b01-8493-62c691216f9c" (UID: "b1236780-03ae-4b01-8493-62c691216f9c"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 00:49:52.362052 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:52.362029 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1236780-03ae-4b01-8493-62c691216f9c-kube-api-access-jzmm7" (OuterVolumeSpecName: "kube-api-access-jzmm7") pod "b1236780-03ae-4b01-8493-62c691216f9c" (UID: "b1236780-03ae-4b01-8493-62c691216f9c"). InnerVolumeSpecName "kube-api-access-jzmm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 00:49:52.461128 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:52.461088 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzmm7\" (UniqueName: \"kubernetes.io/projected/b1236780-03ae-4b01-8493-62c691216f9c-kube-api-access-jzmm7\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:49:52.461128 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:52.461118 2580 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b1236780-03ae-4b01-8493-62c691216f9c-config-file\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:49:52.647151 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:52.647061 2580 generic.go:356] "Generic (PLEG): container finished" podID="b1236780-03ae-4b01-8493-62c691216f9c" containerID="f1e4281dc0c819d669464eef3c4ecadd61434986c730398f7e004bd9c95b29da" exitCode=0 Jul 18 00:49:52.647151 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:52.647110 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-j5tvk" Jul 18 00:49:52.647596 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:52.647150 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-j5tvk" event={"ID":"b1236780-03ae-4b01-8493-62c691216f9c","Type":"ContainerDied","Data":"f1e4281dc0c819d669464eef3c4ecadd61434986c730398f7e004bd9c95b29da"} Jul 18 00:49:52.647596 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:52.647207 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-j5tvk" event={"ID":"b1236780-03ae-4b01-8493-62c691216f9c","Type":"ContainerDied","Data":"4cab5f37dd6c570c660c4e867b3444a887e0b6c40cbd60baf86b9a5b37e8f43e"} Jul 18 00:49:52.647596 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:52.647230 2580 scope.go:117] "RemoveContainer" containerID="f1e4281dc0c819d669464eef3c4ecadd61434986c730398f7e004bd9c95b29da" Jul 18 00:49:52.657285 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:52.657261 2580 scope.go:117] "RemoveContainer" containerID="f1e4281dc0c819d669464eef3c4ecadd61434986c730398f7e004bd9c95b29da" Jul 18 00:49:52.657576 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:49:52.657552 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1e4281dc0c819d669464eef3c4ecadd61434986c730398f7e004bd9c95b29da\": container with ID starting with f1e4281dc0c819d669464eef3c4ecadd61434986c730398f7e004bd9c95b29da not found: ID does not exist" containerID="f1e4281dc0c819d669464eef3c4ecadd61434986c730398f7e004bd9c95b29da" Jul 18 00:49:52.657654 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:52.657588 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1e4281dc0c819d669464eef3c4ecadd61434986c730398f7e004bd9c95b29da"} err="failed to get container status \"f1e4281dc0c819d669464eef3c4ecadd61434986c730398f7e004bd9c95b29da\": rpc error: code = NotFound desc = could not find container \"f1e4281dc0c819d669464eef3c4ecadd61434986c730398f7e004bd9c95b29da\": container with ID starting with f1e4281dc0c819d669464eef3c4ecadd61434986c730398f7e004bd9c95b29da not found: ID does not exist" Jul 18 00:49:52.677374 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:52.677336 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-j5tvk"] Jul 18 00:49:52.680491 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:52.680464 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-j5tvk"] Jul 18 00:49:53.147567 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:53.147526 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-zbzd6"] Jul 18 00:49:53.147975 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:53.147960 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1236780-03ae-4b01-8493-62c691216f9c" containerName="limitador" Jul 18 00:49:53.148034 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:53.147980 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1236780-03ae-4b01-8493-62c691216f9c" containerName="limitador" Jul 18 00:49:53.148073 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:53.148064 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1236780-03ae-4b01-8493-62c691216f9c" containerName="limitador" Jul 18 00:49:53.152624 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:53.152602 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zbzd6" Jul 18 00:49:53.155151 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:53.155127 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 18 00:49:53.155300 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:53.155195 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 18 00:49:53.155300 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:53.155127 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 18 00:49:53.156114 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:53.156096 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-qpj6d\"" Jul 18 00:49:53.159217 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:53.159193 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-zbzd6"] Jul 18 00:49:53.268906 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:53.268867 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjpd5\" (UniqueName: \"kubernetes.io/projected/1d30b4d1-766d-4600-ab4b-3e6851cae623-kube-api-access-zjpd5\") pod \"postgres-69fb4594cf-zbzd6\" (UID: \"1d30b4d1-766d-4600-ab4b-3e6851cae623\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zbzd6" Jul 18 00:49:53.269091 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:53.268995 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1d30b4d1-766d-4600-ab4b-3e6851cae623-data\") pod \"postgres-69fb4594cf-zbzd6\" (UID: \"1d30b4d1-766d-4600-ab4b-3e6851cae623\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zbzd6" Jul 18 00:49:53.370318 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:53.370275 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1d30b4d1-766d-4600-ab4b-3e6851cae623-data\") pod \"postgres-69fb4594cf-zbzd6\" (UID: \"1d30b4d1-766d-4600-ab4b-3e6851cae623\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zbzd6" Jul 18 00:49:53.370318 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:53.370324 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjpd5\" (UniqueName: \"kubernetes.io/projected/1d30b4d1-766d-4600-ab4b-3e6851cae623-kube-api-access-zjpd5\") pod \"postgres-69fb4594cf-zbzd6\" (UID: \"1d30b4d1-766d-4600-ab4b-3e6851cae623\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zbzd6" Jul 18 00:49:53.370783 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:53.370763 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1d30b4d1-766d-4600-ab4b-3e6851cae623-data\") pod \"postgres-69fb4594cf-zbzd6\" (UID: \"1d30b4d1-766d-4600-ab4b-3e6851cae623\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zbzd6" Jul 18 00:49:53.379224 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:53.379203 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjpd5\" (UniqueName: \"kubernetes.io/projected/1d30b4d1-766d-4600-ab4b-3e6851cae623-kube-api-access-zjpd5\") pod \"postgres-69fb4594cf-zbzd6\" (UID: \"1d30b4d1-766d-4600-ab4b-3e6851cae623\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zbzd6" Jul 18 00:49:53.465004 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:53.464975 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zbzd6" Jul 18 00:49:53.605536 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:53.605507 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-zbzd6"] Jul 18 00:49:53.607115 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:49:53.607086 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d30b4d1_766d_4600_ab4b_3e6851cae623.slice/crio-cb2049069b73269812555b7e4e3ff57a8a1d1a39f6172cad6cdb0a0d9b25b6c1 WatchSource:0}: Error finding container cb2049069b73269812555b7e4e3ff57a8a1d1a39f6172cad6cdb0a0d9b25b6c1: Status 404 returned error can't find the container with id cb2049069b73269812555b7e4e3ff57a8a1d1a39f6172cad6cdb0a0d9b25b6c1 Jul 18 00:49:53.654241 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:53.654208 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zbzd6" event={"ID":"1d30b4d1-766d-4600-ab4b-3e6851cae623","Type":"ContainerStarted","Data":"cb2049069b73269812555b7e4e3ff57a8a1d1a39f6172cad6cdb0a0d9b25b6c1"} Jul 18 00:49:54.149313 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:54.149277 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1236780-03ae-4b01-8493-62c691216f9c" path="/var/lib/kubelet/pods/b1236780-03ae-4b01-8493-62c691216f9c/volumes" Jul 18 00:49:58.684701 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:58.684588 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zbzd6" event={"ID":"1d30b4d1-766d-4600-ab4b-3e6851cae623","Type":"ContainerStarted","Data":"e1c1c6a48f2921249d47c98938091ff5ee7914d3b7c070ac5462d5e0944965b1"} Jul 18 00:49:58.685120 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:49:58.684725 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zbzd6" Jul 18 00:50:04.719007 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:04.718979 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zbzd6" Jul 18 00:50:04.735304 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:04.735256 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zbzd6" podStartSLOduration=6.95645013 podStartE2EDuration="11.735241012s" totalImagesPullingTime="4.778790882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:49:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:49:58.700023671 +0000 UTC m=+559.051874022" watchObservedRunningTime="2026-07-18 00:50:04.735241012 +0000 UTC m=+565.087091358" Jul 18 00:50:11.993972 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:11.993939 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6b8b59c454-cs8tw"] Jul 18 00:50:12.196481 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:12.196445 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b8b59c454-cs8tw" Jul 18 00:50:12.199256 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:12.199233 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 18 00:50:12.199402 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:12.199233 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4mw4g\"" Jul 18 00:50:12.199402 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:12.199249 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 18 00:50:12.203030 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:12.203012 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b8b59c454-cs8tw"] Jul 18 00:50:12.203126 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:12.203062 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b7ff4fd55-tmw87"] Jul 18 00:50:12.204751 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:12.204349 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b873cb9-978c-456b-b57f-423a55d1550c-webhook-cert\") pod \"maas-controller-6b8b59c454-cs8tw\" (UID: \"7b873cb9-978c-456b-b57f-423a55d1550c\") " pod="opendatahub/maas-controller-6b8b59c454-cs8tw" Jul 18 00:50:12.204751 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:12.204399 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz8gr\" (UniqueName: \"kubernetes.io/projected/7b873cb9-978c-456b-b57f-423a55d1550c-kube-api-access-tz8gr\") pod \"maas-controller-6b8b59c454-cs8tw\" (UID: \"7b873cb9-978c-456b-b57f-423a55d1550c\") " pod="opendatahub/maas-controller-6b8b59c454-cs8tw" Jul 18 00:50:12.227622 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:12.227593 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b7ff4fd55-tmw87"] Jul 18 00:50:12.227757 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:12.227712 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b7ff4fd55-tmw87" Jul 18 00:50:12.304875 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:12.304788 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtrc9\" (UniqueName: \"kubernetes.io/projected/c9d98bf0-2c41-4158-b33f-56a72c5a92a0-kube-api-access-mtrc9\") pod \"maas-controller-7b7ff4fd55-tmw87\" (UID: \"c9d98bf0-2c41-4158-b33f-56a72c5a92a0\") " pod="opendatahub/maas-controller-7b7ff4fd55-tmw87" Jul 18 00:50:12.304875 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:12.304838 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b873cb9-978c-456b-b57f-423a55d1550c-webhook-cert\") pod \"maas-controller-6b8b59c454-cs8tw\" (UID: \"7b873cb9-978c-456b-b57f-423a55d1550c\") " pod="opendatahub/maas-controller-6b8b59c454-cs8tw" Jul 18 00:50:12.304875 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:12.304873 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz8gr\" (UniqueName: \"kubernetes.io/projected/7b873cb9-978c-456b-b57f-423a55d1550c-kube-api-access-tz8gr\") pod \"maas-controller-6b8b59c454-cs8tw\" (UID: \"7b873cb9-978c-456b-b57f-423a55d1550c\") " pod="opendatahub/maas-controller-6b8b59c454-cs8tw" Jul 18 00:50:12.305160 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:12.304901 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c9d98bf0-2c41-4158-b33f-56a72c5a92a0-webhook-cert\") pod \"maas-controller-7b7ff4fd55-tmw87\" (UID: \"c9d98bf0-2c41-4158-b33f-56a72c5a92a0\") " pod="opendatahub/maas-controller-7b7ff4fd55-tmw87" Jul 18 00:50:12.305160 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:50:12.304985 2580 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 18 00:50:12.305160 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:50:12.305072 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b873cb9-978c-456b-b57f-423a55d1550c-webhook-cert podName:7b873cb9-978c-456b-b57f-423a55d1550c nodeName:}" failed. No retries permitted until 2026-07-18 00:50:12.805050899 +0000 UTC m=+573.156901223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/7b873cb9-978c-456b-b57f-423a55d1550c-webhook-cert") pod "maas-controller-6b8b59c454-cs8tw" (UID: "7b873cb9-978c-456b-b57f-423a55d1550c") : secret "maas-controller-webhook-cert" not found Jul 18 00:50:12.315627 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:12.315596 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz8gr\" (UniqueName: \"kubernetes.io/projected/7b873cb9-978c-456b-b57f-423a55d1550c-kube-api-access-tz8gr\") pod \"maas-controller-6b8b59c454-cs8tw\" (UID: \"7b873cb9-978c-456b-b57f-423a55d1550c\") " pod="opendatahub/maas-controller-6b8b59c454-cs8tw" Jul 18 00:50:12.405820 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:12.405785 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c9d98bf0-2c41-4158-b33f-56a72c5a92a0-webhook-cert\") pod \"maas-controller-7b7ff4fd55-tmw87\" (UID: \"c9d98bf0-2c41-4158-b33f-56a72c5a92a0\") " pod="opendatahub/maas-controller-7b7ff4fd55-tmw87" Jul 18 00:50:12.405986 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:12.405866 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtrc9\" (UniqueName: \"kubernetes.io/projected/c9d98bf0-2c41-4158-b33f-56a72c5a92a0-kube-api-access-mtrc9\") pod \"maas-controller-7b7ff4fd55-tmw87\" (UID: \"c9d98bf0-2c41-4158-b33f-56a72c5a92a0\") " pod="opendatahub/maas-controller-7b7ff4fd55-tmw87" Jul 18 00:50:12.405986 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:50:12.405941 2580 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 18 00:50:12.406073 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:50:12.406027 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9d98bf0-2c41-4158-b33f-56a72c5a92a0-webhook-cert podName:c9d98bf0-2c41-4158-b33f-56a72c5a92a0 nodeName:}" failed. No retries permitted until 2026-07-18 00:50:12.906008976 +0000 UTC m=+573.257859298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/c9d98bf0-2c41-4158-b33f-56a72c5a92a0-webhook-cert") pod "maas-controller-7b7ff4fd55-tmw87" (UID: "c9d98bf0-2c41-4158-b33f-56a72c5a92a0") : secret "maas-controller-webhook-cert" not found Jul 18 00:50:12.415021 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:12.414990 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtrc9\" (UniqueName: \"kubernetes.io/projected/c9d98bf0-2c41-4158-b33f-56a72c5a92a0-kube-api-access-mtrc9\") pod \"maas-controller-7b7ff4fd55-tmw87\" (UID: \"c9d98bf0-2c41-4158-b33f-56a72c5a92a0\") " pod="opendatahub/maas-controller-7b7ff4fd55-tmw87" Jul 18 00:50:12.808555 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:12.808512 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b873cb9-978c-456b-b57f-423a55d1550c-webhook-cert\") pod \"maas-controller-6b8b59c454-cs8tw\" (UID: \"7b873cb9-978c-456b-b57f-423a55d1550c\") " pod="opendatahub/maas-controller-6b8b59c454-cs8tw" Jul 18 00:50:12.808759 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:50:12.808652 2580 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 18 00:50:12.808759 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:50:12.808745 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b873cb9-978c-456b-b57f-423a55d1550c-webhook-cert podName:7b873cb9-978c-456b-b57f-423a55d1550c nodeName:}" failed. No retries permitted until 2026-07-18 00:50:13.808727802 +0000 UTC m=+574.160578124 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/7b873cb9-978c-456b-b57f-423a55d1550c-webhook-cert") pod "maas-controller-6b8b59c454-cs8tw" (UID: "7b873cb9-978c-456b-b57f-423a55d1550c") : secret "maas-controller-webhook-cert" not found Jul 18 00:50:12.909016 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:12.908987 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c9d98bf0-2c41-4158-b33f-56a72c5a92a0-webhook-cert\") pod \"maas-controller-7b7ff4fd55-tmw87\" (UID: \"c9d98bf0-2c41-4158-b33f-56a72c5a92a0\") " pod="opendatahub/maas-controller-7b7ff4fd55-tmw87" Jul 18 00:50:12.911465 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:12.911436 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c9d98bf0-2c41-4158-b33f-56a72c5a92a0-webhook-cert\") pod \"maas-controller-7b7ff4fd55-tmw87\" (UID: \"c9d98bf0-2c41-4158-b33f-56a72c5a92a0\") " pod="opendatahub/maas-controller-7b7ff4fd55-tmw87" Jul 18 00:50:13.137851 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:13.137755 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b7ff4fd55-tmw87" Jul 18 00:50:13.269449 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:13.269422 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b7ff4fd55-tmw87"] Jul 18 00:50:13.271349 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:50:13.271315 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d98bf0_2c41_4158_b33f_56a72c5a92a0.slice/crio-1ecf47e5c64eb6dcba64383cb2ad33e569cf1294ed91ebf5d44992c8633bcffe WatchSource:0}: Error finding container 1ecf47e5c64eb6dcba64383cb2ad33e569cf1294ed91ebf5d44992c8633bcffe: Status 404 returned error can't find the container with id 1ecf47e5c64eb6dcba64383cb2ad33e569cf1294ed91ebf5d44992c8633bcffe Jul 18 00:50:13.758210 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:13.758171 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b7ff4fd55-tmw87" event={"ID":"c9d98bf0-2c41-4158-b33f-56a72c5a92a0","Type":"ContainerStarted","Data":"1ecf47e5c64eb6dcba64383cb2ad33e569cf1294ed91ebf5d44992c8633bcffe"} Jul 18 00:50:13.815664 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:13.815632 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b873cb9-978c-456b-b57f-423a55d1550c-webhook-cert\") pod \"maas-controller-6b8b59c454-cs8tw\" (UID: \"7b873cb9-978c-456b-b57f-423a55d1550c\") " pod="opendatahub/maas-controller-6b8b59c454-cs8tw" Jul 18 00:50:13.818109 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:13.818089 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b873cb9-978c-456b-b57f-423a55d1550c-webhook-cert\") pod \"maas-controller-6b8b59c454-cs8tw\" (UID: \"7b873cb9-978c-456b-b57f-423a55d1550c\") " pod="opendatahub/maas-controller-6b8b59c454-cs8tw" Jul 18 00:50:14.007767 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:14.007728 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b8b59c454-cs8tw" Jul 18 00:50:14.170381 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:14.170354 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b8b59c454-cs8tw"] Jul 18 00:50:14.171814 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:50:14.171792 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b873cb9_978c_456b_b57f_423a55d1550c.slice/crio-6080a3860d520e9263e301f9b1560ac2f0dc524a6004f983671272e60009d1c1 WatchSource:0}: Error finding container 6080a3860d520e9263e301f9b1560ac2f0dc524a6004f983671272e60009d1c1: Status 404 returned error can't find the container with id 6080a3860d520e9263e301f9b1560ac2f0dc524a6004f983671272e60009d1c1 Jul 18 00:50:14.766145 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:14.766067 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b8b59c454-cs8tw" event={"ID":"7b873cb9-978c-456b-b57f-423a55d1550c","Type":"ContainerStarted","Data":"6080a3860d520e9263e301f9b1560ac2f0dc524a6004f983671272e60009d1c1"} Jul 18 00:50:16.778575 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:16.778527 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b7ff4fd55-tmw87" event={"ID":"c9d98bf0-2c41-4158-b33f-56a72c5a92a0","Type":"ContainerStarted","Data":"545a1ab8321ab3ce5d9bcbadfc2fbbfe1a838dccb93dbbc0fb1b4bbcdc1ddfac"} Jul 18 00:50:16.779089 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:16.778628 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b7ff4fd55-tmw87" Jul 18 00:50:16.780172 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:16.780134 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b8b59c454-cs8tw" event={"ID":"7b873cb9-978c-456b-b57f-423a55d1550c","Type":"ContainerStarted","Data":"322d67a4637e6df7da92a57e39798a47d1806664fc0238bbcfbf6e8d1e2226ee"} Jul 18 00:50:16.780328 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:16.780309 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6b8b59c454-cs8tw" Jul 18 00:50:16.800429 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:16.800380 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7b7ff4fd55-tmw87" podStartSLOduration=2.019800706 podStartE2EDuration="4.800367307s" totalImagesPullingTime="2.780566601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:50:16.798427027 +0000 UTC m=+577.150277389" watchObservedRunningTime="2026-07-18 00:50:16.800367307 +0000 UTC m=+577.152217653" Jul 18 00:50:16.814324 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:16.814276 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6b8b59c454-cs8tw" podStartSLOduration=3.929826103 podStartE2EDuration="5.81425752s" totalImagesPullingTime="1.884431417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:50:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:50:16.813803438 +0000 UTC m=+577.165653787" watchObservedRunningTime="2026-07-18 00:50:16.81425752 +0000 UTC m=+577.166107909" Jul 18 00:50:27.788035 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:27.788003 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b7ff4fd55-tmw87" Jul 18 00:50:27.788402 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:27.788071 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6b8b59c454-cs8tw" Jul 18 00:50:27.847240 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:27.847204 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b8b59c454-cs8tw"] Jul 18 00:50:27.847445 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:27.847421 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6b8b59c454-cs8tw" podUID="7b873cb9-978c-456b-b57f-423a55d1550c" containerName="manager" containerID="cri-o://322d67a4637e6df7da92a57e39798a47d1806664fc0238bbcfbf6e8d1e2226ee" gracePeriod=30 Jul 18 00:50:28.085732 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:28.085707 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b8b59c454-cs8tw" Jul 18 00:50:28.142690 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:28.142636 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tz8gr\" (UniqueName: \"kubernetes.io/projected/7b873cb9-978c-456b-b57f-423a55d1550c-kube-api-access-tz8gr\") pod \"7b873cb9-978c-456b-b57f-423a55d1550c\" (UID: \"7b873cb9-978c-456b-b57f-423a55d1550c\") " Jul 18 00:50:28.142854 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:28.142740 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b873cb9-978c-456b-b57f-423a55d1550c-webhook-cert\") pod \"7b873cb9-978c-456b-b57f-423a55d1550c\" (UID: \"7b873cb9-978c-456b-b57f-423a55d1550c\") " Jul 18 00:50:28.144994 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:28.144963 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b873cb9-978c-456b-b57f-423a55d1550c-kube-api-access-tz8gr" (OuterVolumeSpecName: "kube-api-access-tz8gr") pod "7b873cb9-978c-456b-b57f-423a55d1550c" (UID: "7b873cb9-978c-456b-b57f-423a55d1550c"). InnerVolumeSpecName "kube-api-access-tz8gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 00:50:28.145141 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:28.145115 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b873cb9-978c-456b-b57f-423a55d1550c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "7b873cb9-978c-456b-b57f-423a55d1550c" (UID: "7b873cb9-978c-456b-b57f-423a55d1550c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 00:50:28.243594 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:28.243550 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b873cb9-978c-456b-b57f-423a55d1550c-webhook-cert\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:50:28.243594 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:28.243582 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz8gr\" (UniqueName: \"kubernetes.io/projected/7b873cb9-978c-456b-b57f-423a55d1550c-kube-api-access-tz8gr\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:50:28.840586 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:28.840549 2580 generic.go:356] "Generic (PLEG): container finished" podID="7b873cb9-978c-456b-b57f-423a55d1550c" containerID="322d67a4637e6df7da92a57e39798a47d1806664fc0238bbcfbf6e8d1e2226ee" exitCode=0 Jul 18 00:50:28.841051 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:28.840595 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b8b59c454-cs8tw" Jul 18 00:50:28.841051 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:28.840638 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b8b59c454-cs8tw" event={"ID":"7b873cb9-978c-456b-b57f-423a55d1550c","Type":"ContainerDied","Data":"322d67a4637e6df7da92a57e39798a47d1806664fc0238bbcfbf6e8d1e2226ee"} Jul 18 00:50:28.841051 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:28.840708 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b8b59c454-cs8tw" event={"ID":"7b873cb9-978c-456b-b57f-423a55d1550c","Type":"ContainerDied","Data":"6080a3860d520e9263e301f9b1560ac2f0dc524a6004f983671272e60009d1c1"} Jul 18 00:50:28.841051 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:28.840733 2580 scope.go:117] "RemoveContainer" containerID="322d67a4637e6df7da92a57e39798a47d1806664fc0238bbcfbf6e8d1e2226ee" Jul 18 00:50:28.849542 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:28.849316 2580 scope.go:117] "RemoveContainer" containerID="322d67a4637e6df7da92a57e39798a47d1806664fc0238bbcfbf6e8d1e2226ee" Jul 18 00:50:28.849606 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:50:28.849586 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"322d67a4637e6df7da92a57e39798a47d1806664fc0238bbcfbf6e8d1e2226ee\": container with ID starting with 322d67a4637e6df7da92a57e39798a47d1806664fc0238bbcfbf6e8d1e2226ee not found: ID does not exist" containerID="322d67a4637e6df7da92a57e39798a47d1806664fc0238bbcfbf6e8d1e2226ee" Jul 18 00:50:28.849645 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:28.849614 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"322d67a4637e6df7da92a57e39798a47d1806664fc0238bbcfbf6e8d1e2226ee"} err="failed to get container status \"322d67a4637e6df7da92a57e39798a47d1806664fc0238bbcfbf6e8d1e2226ee\": rpc error: code = NotFound desc = could not find container \"322d67a4637e6df7da92a57e39798a47d1806664fc0238bbcfbf6e8d1e2226ee\": container with ID starting with 322d67a4637e6df7da92a57e39798a47d1806664fc0238bbcfbf6e8d1e2226ee not found: ID does not exist" Jul 18 00:50:28.859001 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:28.858977 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b8b59c454-cs8tw"] Jul 18 00:50:28.862380 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:28.862360 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6b8b59c454-cs8tw"] Jul 18 00:50:30.148889 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:30.148857 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b873cb9-978c-456b-b57f-423a55d1550c" path="/var/lib/kubelet/pods/7b873cb9-978c-456b-b57f-423a55d1550c/volumes" Jul 18 00:50:48.479215 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:48.479171 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-775b7f98ff-mgk2l"] Jul 18 00:50:48.479691 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:48.479492 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b873cb9-978c-456b-b57f-423a55d1550c" containerName="manager" Jul 18 00:50:48.479691 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:48.479504 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b873cb9-978c-456b-b57f-423a55d1550c" containerName="manager" Jul 18 00:50:48.479691 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:48.479554 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b873cb9-978c-456b-b57f-423a55d1550c" containerName="manager" Jul 18 00:50:48.482439 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:48.482415 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-775b7f98ff-mgk2l" Jul 18 00:50:48.484701 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:48.484660 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 18 00:50:48.492764 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:48.492741 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-775b7f98ff-mgk2l"] Jul 18 00:50:48.614827 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:48.614790 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b54b1f1-dacb-4bf2-baba-ae004b53990f-maas-api-tls\") pod \"maas-api-775b7f98ff-mgk2l\" (UID: \"6b54b1f1-dacb-4bf2-baba-ae004b53990f\") " pod="odh-ai-gateway-infra/maas-api-775b7f98ff-mgk2l" Jul 18 00:50:48.614827 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:48.614828 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4wrk\" (UniqueName: \"kubernetes.io/projected/6b54b1f1-dacb-4bf2-baba-ae004b53990f-kube-api-access-c4wrk\") pod \"maas-api-775b7f98ff-mgk2l\" (UID: \"6b54b1f1-dacb-4bf2-baba-ae004b53990f\") " pod="odh-ai-gateway-infra/maas-api-775b7f98ff-mgk2l" Jul 18 00:50:48.615041 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:48.614901 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b54b1f1-dacb-4bf2-baba-ae004b53990f-openshift-service-ca\") pod \"maas-api-775b7f98ff-mgk2l\" (UID: \"6b54b1f1-dacb-4bf2-baba-ae004b53990f\") " pod="odh-ai-gateway-infra/maas-api-775b7f98ff-mgk2l" Jul 18 00:50:48.716148 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:48.716106 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b54b1f1-dacb-4bf2-baba-ae004b53990f-maas-api-tls\") pod \"maas-api-775b7f98ff-mgk2l\" (UID: \"6b54b1f1-dacb-4bf2-baba-ae004b53990f\") " pod="odh-ai-gateway-infra/maas-api-775b7f98ff-mgk2l" Jul 18 00:50:48.716148 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:48.716146 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4wrk\" (UniqueName: \"kubernetes.io/projected/6b54b1f1-dacb-4bf2-baba-ae004b53990f-kube-api-access-c4wrk\") pod \"maas-api-775b7f98ff-mgk2l\" (UID: \"6b54b1f1-dacb-4bf2-baba-ae004b53990f\") " pod="odh-ai-gateway-infra/maas-api-775b7f98ff-mgk2l" Jul 18 00:50:48.716375 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:48.716203 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b54b1f1-dacb-4bf2-baba-ae004b53990f-openshift-service-ca\") pod \"maas-api-775b7f98ff-mgk2l\" (UID: \"6b54b1f1-dacb-4bf2-baba-ae004b53990f\") " pod="odh-ai-gateway-infra/maas-api-775b7f98ff-mgk2l" Jul 18 00:50:48.716913 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:48.716889 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b54b1f1-dacb-4bf2-baba-ae004b53990f-openshift-service-ca\") pod \"maas-api-775b7f98ff-mgk2l\" (UID: \"6b54b1f1-dacb-4bf2-baba-ae004b53990f\") " pod="odh-ai-gateway-infra/maas-api-775b7f98ff-mgk2l" Jul 18 00:50:48.718698 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:48.718659 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b54b1f1-dacb-4bf2-baba-ae004b53990f-maas-api-tls\") pod \"maas-api-775b7f98ff-mgk2l\" (UID: \"6b54b1f1-dacb-4bf2-baba-ae004b53990f\") " pod="odh-ai-gateway-infra/maas-api-775b7f98ff-mgk2l" Jul 18 00:50:48.723749 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:48.723732 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4wrk\" (UniqueName: \"kubernetes.io/projected/6b54b1f1-dacb-4bf2-baba-ae004b53990f-kube-api-access-c4wrk\") pod \"maas-api-775b7f98ff-mgk2l\" (UID: \"6b54b1f1-dacb-4bf2-baba-ae004b53990f\") " pod="odh-ai-gateway-infra/maas-api-775b7f98ff-mgk2l" Jul 18 00:50:48.794262 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:48.794178 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-775b7f98ff-mgk2l" Jul 18 00:50:48.933962 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:48.933816 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-775b7f98ff-mgk2l"] Jul 18 00:50:48.936443 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:50:48.936412 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b54b1f1_dacb_4bf2_baba_ae004b53990f.slice/crio-b106d9d49bbbefa6a4cd61ee1520810a39778d2ec29edf063d7ec18ce53a11fe WatchSource:0}: Error finding container b106d9d49bbbefa6a4cd61ee1520810a39778d2ec29edf063d7ec18ce53a11fe: Status 404 returned error can't find the container with id b106d9d49bbbefa6a4cd61ee1520810a39778d2ec29edf063d7ec18ce53a11fe Jul 18 00:50:48.937726 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:48.937710 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 00:50:48.954002 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:48.953974 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-775b7f98ff-mgk2l" event={"ID":"6b54b1f1-dacb-4bf2-baba-ae004b53990f","Type":"ContainerStarted","Data":"b106d9d49bbbefa6a4cd61ee1520810a39778d2ec29edf063d7ec18ce53a11fe"} Jul 18 00:50:49.139222 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:49.139142 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-jcklw"] Jul 18 00:50:49.145695 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:49.145653 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-jcklw" Jul 18 00:50:49.148163 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:49.148138 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-cxzj8\"" Jul 18 00:50:49.148281 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:49.148139 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 18 00:50:49.153426 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:49.153401 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-jcklw"] Jul 18 00:50:49.220525 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:49.220490 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsbtr\" (UniqueName: \"kubernetes.io/projected/734a2b3a-9cfa-491c-a66f-c9913a167e6c-kube-api-access-dsbtr\") pod \"payload-processing-5bf47cd988-jcklw\" (UID: \"734a2b3a-9cfa-491c-a66f-c9913a167e6c\") " pod="openshift-ingress/payload-processing-5bf47cd988-jcklw" Jul 18 00:50:49.220777 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:49.220628 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/734a2b3a-9cfa-491c-a66f-c9913a167e6c-plugins-config-volume\") pod \"payload-processing-5bf47cd988-jcklw\" (UID: \"734a2b3a-9cfa-491c-a66f-c9913a167e6c\") " pod="openshift-ingress/payload-processing-5bf47cd988-jcklw" Jul 18 00:50:49.321640 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:49.321589 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/734a2b3a-9cfa-491c-a66f-c9913a167e6c-plugins-config-volume\") pod \"payload-processing-5bf47cd988-jcklw\" (UID: \"734a2b3a-9cfa-491c-a66f-c9913a167e6c\") " pod="openshift-ingress/payload-processing-5bf47cd988-jcklw" Jul 18 00:50:49.321817 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:49.321715 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsbtr\" (UniqueName: \"kubernetes.io/projected/734a2b3a-9cfa-491c-a66f-c9913a167e6c-kube-api-access-dsbtr\") pod \"payload-processing-5bf47cd988-jcklw\" (UID: \"734a2b3a-9cfa-491c-a66f-c9913a167e6c\") " pod="openshift-ingress/payload-processing-5bf47cd988-jcklw" Jul 18 00:50:49.322420 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:49.322401 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/734a2b3a-9cfa-491c-a66f-c9913a167e6c-plugins-config-volume\") pod \"payload-processing-5bf47cd988-jcklw\" (UID: \"734a2b3a-9cfa-491c-a66f-c9913a167e6c\") " pod="openshift-ingress/payload-processing-5bf47cd988-jcklw" Jul 18 00:50:49.335056 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:49.335021 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsbtr\" (UniqueName: \"kubernetes.io/projected/734a2b3a-9cfa-491c-a66f-c9913a167e6c-kube-api-access-dsbtr\") pod \"payload-processing-5bf47cd988-jcklw\" (UID: \"734a2b3a-9cfa-491c-a66f-c9913a167e6c\") " pod="openshift-ingress/payload-processing-5bf47cd988-jcklw" Jul 18 00:50:49.459397 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:49.459321 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-jcklw" Jul 18 00:50:49.618913 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:49.618881 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-jcklw"] Jul 18 00:50:49.622435 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:50:49.622393 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734a2b3a_9cfa_491c_a66f_c9913a167e6c.slice/crio-a1ddf6aaf3372f147f5197d4ecffcd6e8b7052fc926bad5f5e9796bac700f369 WatchSource:0}: Error finding container a1ddf6aaf3372f147f5197d4ecffcd6e8b7052fc926bad5f5e9796bac700f369: Status 404 returned error can't find the container with id a1ddf6aaf3372f147f5197d4ecffcd6e8b7052fc926bad5f5e9796bac700f369 Jul 18 00:50:49.961755 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:49.961718 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-jcklw" event={"ID":"734a2b3a-9cfa-491c-a66f-c9913a167e6c","Type":"ContainerStarted","Data":"a1ddf6aaf3372f147f5197d4ecffcd6e8b7052fc926bad5f5e9796bac700f369"} Jul 18 00:50:50.968957 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:50.968919 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-775b7f98ff-mgk2l" event={"ID":"6b54b1f1-dacb-4bf2-baba-ae004b53990f","Type":"ContainerStarted","Data":"e73b6ac5c5a334746b0bbdc7c3e6277aab54f6b39a4655f4ebf10c4b535db51d"} Jul 18 00:50:50.969500 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:50.969010 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-775b7f98ff-mgk2l" Jul 18 00:50:50.986148 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:50.986089 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-775b7f98ff-mgk2l" podStartSLOduration=1.6805565919999998 podStartE2EDuration="2.98606853s" totalImagesPullingTime="1.305511938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:50:50.984836007 +0000 UTC m=+611.336686353" watchObservedRunningTime="2026-07-18 00:50:50.98606853 +0000 UTC m=+611.337918877" Jul 18 00:50:51.975625 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:51.975582 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-jcklw" event={"ID":"734a2b3a-9cfa-491c-a66f-c9913a167e6c","Type":"ContainerStarted","Data":"6d8dabee951fb9c58b2e25f3f3704deea776b692299ac24a49f2af5755ad9357"} Jul 18 00:50:51.976037 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:51.975867 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-jcklw" Jul 18 00:50:51.992064 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:51.992017 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-jcklw" podStartSLOduration=1.267494425 podStartE2EDuration="2.992002075s" totalImagesPullingTime="1.72450765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:50:51.98977275 +0000 UTC m=+612.341623117" watchObservedRunningTime="2026-07-18 00:50:51.992002075 +0000 UTC m=+612.343852420" Jul 18 00:50:56.981356 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:50:56.981327 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-775b7f98ff-mgk2l" Jul 18 00:51:01.816742 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:01.816619 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57"] Jul 18 00:51:01.820327 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:01.820307 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57" Jul 18 00:51:01.822942 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:01.822913 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 18 00:51:01.823946 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:01.823920 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-cdtw4\"" Jul 18 00:51:01.824068 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:01.823993 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 18 00:51:01.824118 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:01.823922 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 18 00:51:01.831523 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:01.831500 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57"] Jul 18 00:51:01.932041 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:01.932005 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a1845e9-c79e-47a8-a4fa-779f299ff699-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57\" (UID: \"7a1845e9-c79e-47a8-a4fa-779f299ff699\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57" Jul 18 00:51:01.932041 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:01.932044 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a1845e9-c79e-47a8-a4fa-779f299ff699-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57\" (UID: \"7a1845e9-c79e-47a8-a4fa-779f299ff699\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57" Jul 18 00:51:01.932270 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:01.932108 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a1845e9-c79e-47a8-a4fa-779f299ff699-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57\" (UID: \"7a1845e9-c79e-47a8-a4fa-779f299ff699\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57" Jul 18 00:51:01.932270 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:01.932135 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a1845e9-c79e-47a8-a4fa-779f299ff699-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57\" (UID: \"7a1845e9-c79e-47a8-a4fa-779f299ff699\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57" Jul 18 00:51:01.932270 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:01.932164 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhv85\" (UniqueName: \"kubernetes.io/projected/7a1845e9-c79e-47a8-a4fa-779f299ff699-kube-api-access-lhv85\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57\" (UID: \"7a1845e9-c79e-47a8-a4fa-779f299ff699\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57" Jul 18 00:51:01.932378 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:01.932267 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a1845e9-c79e-47a8-a4fa-779f299ff699-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57\" (UID: \"7a1845e9-c79e-47a8-a4fa-779f299ff699\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57" Jul 18 00:51:02.033261 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:02.033225 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a1845e9-c79e-47a8-a4fa-779f299ff699-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57\" (UID: \"7a1845e9-c79e-47a8-a4fa-779f299ff699\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57" Jul 18 00:51:02.033425 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:02.033284 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a1845e9-c79e-47a8-a4fa-779f299ff699-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57\" (UID: \"7a1845e9-c79e-47a8-a4fa-779f299ff699\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57" Jul 18 00:51:02.033425 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:02.033312 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a1845e9-c79e-47a8-a4fa-779f299ff699-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57\" (UID: \"7a1845e9-c79e-47a8-a4fa-779f299ff699\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57" Jul 18 00:51:02.033425 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:02.033349 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a1845e9-c79e-47a8-a4fa-779f299ff699-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57\" (UID: \"7a1845e9-c79e-47a8-a4fa-779f299ff699\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57" Jul 18 00:51:02.033425 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:02.033370 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a1845e9-c79e-47a8-a4fa-779f299ff699-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57\" (UID: \"7a1845e9-c79e-47a8-a4fa-779f299ff699\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57" Jul 18 00:51:02.033425 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:02.033393 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhv85\" (UniqueName: \"kubernetes.io/projected/7a1845e9-c79e-47a8-a4fa-779f299ff699-kube-api-access-lhv85\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57\" (UID: \"7a1845e9-c79e-47a8-a4fa-779f299ff699\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57" Jul 18 00:51:02.033828 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:02.033802 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a1845e9-c79e-47a8-a4fa-779f299ff699-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57\" (UID: \"7a1845e9-c79e-47a8-a4fa-779f299ff699\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57" Jul 18 00:51:02.033901 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:02.033838 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a1845e9-c79e-47a8-a4fa-779f299ff699-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57\" (UID: \"7a1845e9-c79e-47a8-a4fa-779f299ff699\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57" Jul 18 00:51:02.033901 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:02.033851 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a1845e9-c79e-47a8-a4fa-779f299ff699-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57\" (UID: \"7a1845e9-c79e-47a8-a4fa-779f299ff699\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57" Jul 18 00:51:02.035922 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:02.035899 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a1845e9-c79e-47a8-a4fa-779f299ff699-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57\" (UID: \"7a1845e9-c79e-47a8-a4fa-779f299ff699\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57" Jul 18 00:51:02.036024 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:02.035965 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a1845e9-c79e-47a8-a4fa-779f299ff699-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57\" (UID: \"7a1845e9-c79e-47a8-a4fa-779f299ff699\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57" Jul 18 00:51:02.041532 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:02.041506 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhv85\" (UniqueName: \"kubernetes.io/projected/7a1845e9-c79e-47a8-a4fa-779f299ff699-kube-api-access-lhv85\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57\" (UID: \"7a1845e9-c79e-47a8-a4fa-779f299ff699\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57" Jul 18 00:51:02.132060 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:02.131953 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57" Jul 18 00:51:02.271762 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:02.271736 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57"] Jul 18 00:51:02.273733 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:51:02.273707 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a1845e9_c79e_47a8_a4fa_779f299ff699.slice/crio-151561de021387aee854667a2a107a6239fd41fbddae40c44be6601dfe340499 WatchSource:0}: Error finding container 151561de021387aee854667a2a107a6239fd41fbddae40c44be6601dfe340499: Status 404 returned error can't find the container with id 151561de021387aee854667a2a107a6239fd41fbddae40c44be6601dfe340499 Jul 18 00:51:02.988535 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:02.988507 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-jcklw" Jul 18 00:51:03.036202 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:03.036153 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57" event={"ID":"7a1845e9-c79e-47a8-a4fa-779f299ff699","Type":"ContainerStarted","Data":"151561de021387aee854667a2a107a6239fd41fbddae40c44be6601dfe340499"} Jul 18 00:51:04.042742 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:04.042705 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57" event={"ID":"7a1845e9-c79e-47a8-a4fa-779f299ff699","Type":"ContainerStarted","Data":"3ffff14b01172d64b12d738ded06af3f7ec7484364b32b0b7a1e1eff4e9cffde"} Jul 18 00:51:04.063472 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:04.063417 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57" podStartSLOduration=1.957913502 podStartE2EDuration="3.063400602s" totalImagesPullingTime="1.1054871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:51:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:51:04.060543822 +0000 UTC m=+624.412394173" watchObservedRunningTime="2026-07-18 00:51:04.063400602 +0000 UTC m=+624.415250947" Jul 18 00:51:12.132183 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:12.132152 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57" Jul 18 00:51:12.132585 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:12.132221 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57" Jul 18 00:51:12.149195 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:12.149172 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57" Jul 18 00:51:13.096902 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:13.096867 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57" Jul 18 00:51:46.921952 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:46.921897 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tgpsg"] Jul 18 00:51:46.928242 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:46.928220 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgpsg" Jul 18 00:51:46.930836 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:46.930812 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 18 00:51:46.934534 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:46.934472 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tgpsg"] Jul 18 00:51:46.936298 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:46.936201 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-659v7\" (UniqueName: \"kubernetes.io/projected/a4aba52a-c110-4efd-8b66-5a806a4365c8-kube-api-access-659v7\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgpsg\" (UID: \"a4aba52a-c110-4efd-8b66-5a806a4365c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgpsg" Jul 18 00:51:46.936429 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:46.936332 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4aba52a-c110-4efd-8b66-5a806a4365c8-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgpsg\" (UID: \"a4aba52a-c110-4efd-8b66-5a806a4365c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgpsg" Jul 18 00:51:46.936429 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:46.936399 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4aba52a-c110-4efd-8b66-5a806a4365c8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgpsg\" (UID: \"a4aba52a-c110-4efd-8b66-5a806a4365c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgpsg" Jul 18 00:51:46.936552 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:46.936431 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4aba52a-c110-4efd-8b66-5a806a4365c8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgpsg\" (UID: \"a4aba52a-c110-4efd-8b66-5a806a4365c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgpsg" Jul 18 00:51:46.936552 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:46.936467 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4aba52a-c110-4efd-8b66-5a806a4365c8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgpsg\" (UID: \"a4aba52a-c110-4efd-8b66-5a806a4365c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgpsg" Jul 18 00:51:46.936655 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:46.936548 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4aba52a-c110-4efd-8b66-5a806a4365c8-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgpsg\" (UID: \"a4aba52a-c110-4efd-8b66-5a806a4365c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgpsg" Jul 18 00:51:47.037179 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:47.037140 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-659v7\" (UniqueName: \"kubernetes.io/projected/a4aba52a-c110-4efd-8b66-5a806a4365c8-kube-api-access-659v7\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgpsg\" (UID: \"a4aba52a-c110-4efd-8b66-5a806a4365c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgpsg" Jul 18 00:51:47.037354 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:47.037195 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4aba52a-c110-4efd-8b66-5a806a4365c8-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgpsg\" (UID: \"a4aba52a-c110-4efd-8b66-5a806a4365c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgpsg" Jul 18 00:51:47.037354 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:47.037223 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4aba52a-c110-4efd-8b66-5a806a4365c8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgpsg\" (UID: \"a4aba52a-c110-4efd-8b66-5a806a4365c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgpsg" Jul 18 00:51:47.037354 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:47.037240 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4aba52a-c110-4efd-8b66-5a806a4365c8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgpsg\" (UID: \"a4aba52a-c110-4efd-8b66-5a806a4365c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgpsg" Jul 18 00:51:47.037354 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:47.037261 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4aba52a-c110-4efd-8b66-5a806a4365c8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgpsg\" (UID: \"a4aba52a-c110-4efd-8b66-5a806a4365c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgpsg" Jul 18 00:51:47.037354 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:47.037285 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4aba52a-c110-4efd-8b66-5a806a4365c8-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgpsg\" (UID: \"a4aba52a-c110-4efd-8b66-5a806a4365c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgpsg" Jul 18 00:51:47.037641 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:47.037620 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4aba52a-c110-4efd-8b66-5a806a4365c8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgpsg\" (UID: \"a4aba52a-c110-4efd-8b66-5a806a4365c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgpsg" Jul 18 00:51:47.037725 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:47.037643 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4aba52a-c110-4efd-8b66-5a806a4365c8-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgpsg\" (UID: \"a4aba52a-c110-4efd-8b66-5a806a4365c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgpsg" Jul 18 00:51:47.037783 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:47.037763 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4aba52a-c110-4efd-8b66-5a806a4365c8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgpsg\" (UID: \"a4aba52a-c110-4efd-8b66-5a806a4365c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgpsg" Jul 18 00:51:47.039846 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:47.039820 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4aba52a-c110-4efd-8b66-5a806a4365c8-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgpsg\" (UID: \"a4aba52a-c110-4efd-8b66-5a806a4365c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgpsg" Jul 18 00:51:47.040073 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:47.040054 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4aba52a-c110-4efd-8b66-5a806a4365c8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgpsg\" (UID: \"a4aba52a-c110-4efd-8b66-5a806a4365c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgpsg" Jul 18 00:51:47.045030 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:47.045007 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-659v7\" (UniqueName: \"kubernetes.io/projected/a4aba52a-c110-4efd-8b66-5a806a4365c8-kube-api-access-659v7\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgpsg\" (UID: \"a4aba52a-c110-4efd-8b66-5a806a4365c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgpsg" Jul 18 00:51:47.239905 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:47.239871 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgpsg" Jul 18 00:51:47.375945 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:47.375916 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tgpsg"] Jul 18 00:51:47.377402 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:51:47.377379 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4aba52a_c110_4efd_8b66_5a806a4365c8.slice/crio-d0c37f1b279ee36aafc2656a34399baecd7b05d664538154f8969590d5e7a0ac WatchSource:0}: Error finding container d0c37f1b279ee36aafc2656a34399baecd7b05d664538154f8969590d5e7a0ac: Status 404 returned error can't find the container with id d0c37f1b279ee36aafc2656a34399baecd7b05d664538154f8969590d5e7a0ac Jul 18 00:51:48.271370 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:48.271336 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgpsg" event={"ID":"a4aba52a-c110-4efd-8b66-5a806a4365c8","Type":"ContainerStarted","Data":"9e7562d2a974e0b2a304a8374c44fb42d432dbd9d09fe5d522d3cd43ba15fcaf"} Jul 18 00:51:48.271795 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:48.271388 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgpsg" event={"ID":"a4aba52a-c110-4efd-8b66-5a806a4365c8","Type":"ContainerStarted","Data":"d0c37f1b279ee36aafc2656a34399baecd7b05d664538154f8969590d5e7a0ac"} Jul 18 00:51:48.297480 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:48.297429 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgpsg" podStartSLOduration=2.120957279 podStartE2EDuration="2.297413815s" totalImagesPullingTime="176.456536ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:51:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:51:48.295076122 +0000 UTC m=+668.646926467" watchObservedRunningTime="2026-07-18 00:51:48.297413815 +0000 UTC m=+668.649264188" Jul 18 00:51:57.240494 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:57.240458 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgpsg" Jul 18 00:51:57.241017 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:57.240525 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgpsg" Jul 18 00:51:57.252905 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:57.252877 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgpsg" Jul 18 00:51:57.326182 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:51:57.326153 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgpsg" Jul 18 00:52:07.613608 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:52:07.613570 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-w45pv"] Jul 18 00:52:07.616956 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:52:07.616940 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w45pv" Jul 18 00:52:07.619550 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:52:07.619530 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 18 00:52:07.630404 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:52:07.630382 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-w45pv"] Jul 18 00:52:07.714743 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:52:07.714703 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05b8e38e-1881-4eb7-b299-030868877fc9-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w45pv\" (UID: \"05b8e38e-1881-4eb7-b299-030868877fc9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w45pv" Jul 18 00:52:07.714926 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:52:07.714768 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05b8e38e-1881-4eb7-b299-030868877fc9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w45pv\" (UID: \"05b8e38e-1881-4eb7-b299-030868877fc9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w45pv" Jul 18 00:52:07.714926 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:52:07.714799 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gvpk\" (UniqueName: \"kubernetes.io/projected/05b8e38e-1881-4eb7-b299-030868877fc9-kube-api-access-8gvpk\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w45pv\" (UID: \"05b8e38e-1881-4eb7-b299-030868877fc9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w45pv" Jul 18 00:52:07.714926 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:52:07.714880 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05b8e38e-1881-4eb7-b299-030868877fc9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w45pv\" (UID: \"05b8e38e-1881-4eb7-b299-030868877fc9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w45pv" Jul 18 00:52:07.715086 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:52:07.714948 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05b8e38e-1881-4eb7-b299-030868877fc9-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w45pv\" (UID: \"05b8e38e-1881-4eb7-b299-030868877fc9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w45pv" Jul 18 00:52:07.715086 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:52:07.714986 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05b8e38e-1881-4eb7-b299-030868877fc9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w45pv\" (UID: \"05b8e38e-1881-4eb7-b299-030868877fc9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w45pv" Jul 18 00:52:07.815961 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:52:07.815922 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05b8e38e-1881-4eb7-b299-030868877fc9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w45pv\" (UID: \"05b8e38e-1881-4eb7-b299-030868877fc9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w45pv" Jul 18 00:52:07.816182 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:52:07.815974 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05b8e38e-1881-4eb7-b299-030868877fc9-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w45pv\" (UID: \"05b8e38e-1881-4eb7-b299-030868877fc9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w45pv" Jul 18 00:52:07.816182 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:52:07.815999 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05b8e38e-1881-4eb7-b299-030868877fc9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w45pv\" (UID: \"05b8e38e-1881-4eb7-b299-030868877fc9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w45pv" Jul 18 00:52:07.816182 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:52:07.816036 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05b8e38e-1881-4eb7-b299-030868877fc9-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w45pv\" (UID: \"05b8e38e-1881-4eb7-b299-030868877fc9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w45pv" Jul 18 00:52:07.816182 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:52:07.816087 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05b8e38e-1881-4eb7-b299-030868877fc9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w45pv\" (UID: \"05b8e38e-1881-4eb7-b299-030868877fc9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w45pv" Jul 18 00:52:07.816182 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:52:07.816111 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gvpk\" (UniqueName: \"kubernetes.io/projected/05b8e38e-1881-4eb7-b299-030868877fc9-kube-api-access-8gvpk\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w45pv\" (UID: \"05b8e38e-1881-4eb7-b299-030868877fc9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w45pv" Jul 18 00:52:07.816514 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:52:07.816386 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05b8e38e-1881-4eb7-b299-030868877fc9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w45pv\" (UID: \"05b8e38e-1881-4eb7-b299-030868877fc9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w45pv" Jul 18 00:52:07.816562 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:52:07.816529 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05b8e38e-1881-4eb7-b299-030868877fc9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w45pv\" (UID: \"05b8e38e-1881-4eb7-b299-030868877fc9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w45pv" Jul 18 00:52:07.816730 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:52:07.816709 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05b8e38e-1881-4eb7-b299-030868877fc9-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w45pv\" (UID: \"05b8e38e-1881-4eb7-b299-030868877fc9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w45pv" Jul 18 00:52:07.818438 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:52:07.818415 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05b8e38e-1881-4eb7-b299-030868877fc9-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w45pv\" (UID: \"05b8e38e-1881-4eb7-b299-030868877fc9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w45pv" Jul 18 00:52:07.818657 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:52:07.818638 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05b8e38e-1881-4eb7-b299-030868877fc9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w45pv\" (UID: \"05b8e38e-1881-4eb7-b299-030868877fc9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w45pv" Jul 18 00:52:07.829463 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:52:07.829440 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gvpk\" (UniqueName: \"kubernetes.io/projected/05b8e38e-1881-4eb7-b299-030868877fc9-kube-api-access-8gvpk\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w45pv\" (UID: \"05b8e38e-1881-4eb7-b299-030868877fc9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w45pv" Jul 18 00:52:07.927425 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:52:07.927342 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w45pv" Jul 18 00:52:08.063488 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:52:08.063464 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-w45pv"] Jul 18 00:52:08.065470 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:52:08.065442 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b8e38e_1881_4eb7_b299_030868877fc9.slice/crio-c756fc4beb44d81d1d515c23986ad18d99e35755ad4adaeacb1fd549bd0094dd WatchSource:0}: Error finding container c756fc4beb44d81d1d515c23986ad18d99e35755ad4adaeacb1fd549bd0094dd: Status 404 returned error can't find the container with id c756fc4beb44d81d1d515c23986ad18d99e35755ad4adaeacb1fd549bd0094dd Jul 18 00:52:08.369802 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:52:08.369761 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w45pv" event={"ID":"05b8e38e-1881-4eb7-b299-030868877fc9","Type":"ContainerStarted","Data":"62c203ea6fda0b550dfe254cc7736f8dd12ed15691ff57e596f8da696494ad58"} Jul 18 00:52:08.369989 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:52:08.369833 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w45pv" event={"ID":"05b8e38e-1881-4eb7-b299-030868877fc9","Type":"ContainerStarted","Data":"c756fc4beb44d81d1d515c23986ad18d99e35755ad4adaeacb1fd549bd0094dd"} Jul 18 00:52:08.387124 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:52:08.387077 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w45pv" podStartSLOduration=1.21449113 podStartE2EDuration="1.387061285s" totalImagesPullingTime="172.570155ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:52:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:52:08.385257315 +0000 UTC m=+688.737107663" watchObservedRunningTime="2026-07-18 00:52:08.387061285 +0000 UTC m=+688.738911630" Jul 18 00:52:17.927490 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:52:17.927450 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w45pv" Jul 18 00:52:17.927919 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:52:17.927509 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w45pv" Jul 18 00:52:17.940535 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:52:17.940508 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w45pv" Jul 18 00:52:18.429228 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:52:18.429200 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w45pv" Jul 18 00:54:21.596167 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:54:21.596081 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b7ff4fd55-tmw87"] Jul 18 00:54:21.596650 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:54:21.596294 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7b7ff4fd55-tmw87" podUID="c9d98bf0-2c41-4158-b33f-56a72c5a92a0" containerName="manager" containerID="cri-o://545a1ab8321ab3ce5d9bcbadfc2fbbfe1a838dccb93dbbc0fb1b4bbcdc1ddfac" gracePeriod=30 Jul 18 00:54:21.841252 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:54:21.841221 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b7ff4fd55-tmw87" Jul 18 00:54:21.916769 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:54:21.916659 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtrc9\" (UniqueName: \"kubernetes.io/projected/c9d98bf0-2c41-4158-b33f-56a72c5a92a0-kube-api-access-mtrc9\") pod \"c9d98bf0-2c41-4158-b33f-56a72c5a92a0\" (UID: \"c9d98bf0-2c41-4158-b33f-56a72c5a92a0\") " Jul 18 00:54:21.916769 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:54:21.916737 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c9d98bf0-2c41-4158-b33f-56a72c5a92a0-webhook-cert\") pod \"c9d98bf0-2c41-4158-b33f-56a72c5a92a0\" (UID: \"c9d98bf0-2c41-4158-b33f-56a72c5a92a0\") " Jul 18 00:54:21.918911 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:54:21.918878 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9d98bf0-2c41-4158-b33f-56a72c5a92a0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c9d98bf0-2c41-4158-b33f-56a72c5a92a0" (UID: "c9d98bf0-2c41-4158-b33f-56a72c5a92a0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 00:54:21.918911 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:54:21.918894 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d98bf0-2c41-4158-b33f-56a72c5a92a0-kube-api-access-mtrc9" (OuterVolumeSpecName: "kube-api-access-mtrc9") pod "c9d98bf0-2c41-4158-b33f-56a72c5a92a0" (UID: "c9d98bf0-2c41-4158-b33f-56a72c5a92a0"). InnerVolumeSpecName "kube-api-access-mtrc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 00:54:22.018330 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:54:22.018280 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtrc9\" (UniqueName: \"kubernetes.io/projected/c9d98bf0-2c41-4158-b33f-56a72c5a92a0-kube-api-access-mtrc9\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:54:22.018330 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:54:22.018325 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c9d98bf0-2c41-4158-b33f-56a72c5a92a0-webhook-cert\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:54:22.024764 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:54:22.024734 2580 generic.go:356] "Generic (PLEG): container finished" podID="c9d98bf0-2c41-4158-b33f-56a72c5a92a0" containerID="545a1ab8321ab3ce5d9bcbadfc2fbbfe1a838dccb93dbbc0fb1b4bbcdc1ddfac" exitCode=0 Jul 18 00:54:22.024898 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:54:22.024776 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b7ff4fd55-tmw87" Jul 18 00:54:22.024898 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:54:22.024789 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b7ff4fd55-tmw87" event={"ID":"c9d98bf0-2c41-4158-b33f-56a72c5a92a0","Type":"ContainerDied","Data":"545a1ab8321ab3ce5d9bcbadfc2fbbfe1a838dccb93dbbc0fb1b4bbcdc1ddfac"} Jul 18 00:54:22.024898 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:54:22.024840 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b7ff4fd55-tmw87" event={"ID":"c9d98bf0-2c41-4158-b33f-56a72c5a92a0","Type":"ContainerDied","Data":"1ecf47e5c64eb6dcba64383cb2ad33e569cf1294ed91ebf5d44992c8633bcffe"} Jul 18 00:54:22.024898 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:54:22.024865 2580 scope.go:117] "RemoveContainer" containerID="545a1ab8321ab3ce5d9bcbadfc2fbbfe1a838dccb93dbbc0fb1b4bbcdc1ddfac" Jul 18 00:54:22.034763 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:54:22.034701 2580 scope.go:117] "RemoveContainer" containerID="545a1ab8321ab3ce5d9bcbadfc2fbbfe1a838dccb93dbbc0fb1b4bbcdc1ddfac" Jul 18 00:54:22.035123 ip-10-0-134-239 kubenswrapper[2580]: E0718 00:54:22.035100 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"545a1ab8321ab3ce5d9bcbadfc2fbbfe1a838dccb93dbbc0fb1b4bbcdc1ddfac\": container with ID starting with 545a1ab8321ab3ce5d9bcbadfc2fbbfe1a838dccb93dbbc0fb1b4bbcdc1ddfac not found: ID does not exist" containerID="545a1ab8321ab3ce5d9bcbadfc2fbbfe1a838dccb93dbbc0fb1b4bbcdc1ddfac" Jul 18 00:54:22.035213 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:54:22.035130 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"545a1ab8321ab3ce5d9bcbadfc2fbbfe1a838dccb93dbbc0fb1b4bbcdc1ddfac"} err="failed to get container status \"545a1ab8321ab3ce5d9bcbadfc2fbbfe1a838dccb93dbbc0fb1b4bbcdc1ddfac\": rpc error: code = NotFound desc = could not find container \"545a1ab8321ab3ce5d9bcbadfc2fbbfe1a838dccb93dbbc0fb1b4bbcdc1ddfac\": container with ID starting with 545a1ab8321ab3ce5d9bcbadfc2fbbfe1a838dccb93dbbc0fb1b4bbcdc1ddfac not found: ID does not exist" Jul 18 00:54:22.047065 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:54:22.047036 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b7ff4fd55-tmw87"] Jul 18 00:54:22.050216 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:54:22.050187 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7b7ff4fd55-tmw87"] Jul 18 00:54:22.150354 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:54:22.150311 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9d98bf0-2c41-4158-b33f-56a72c5a92a0" path="/var/lib/kubelet/pods/c9d98bf0-2c41-4158-b33f-56a72c5a92a0/volumes" Jul 18 00:55:22.829269 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:22.829233 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b7ff4fd55-nv7nj"] Jul 18 00:55:22.829698 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:22.829559 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9d98bf0-2c41-4158-b33f-56a72c5a92a0" containerName="manager" Jul 18 00:55:22.829698 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:22.829571 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9d98bf0-2c41-4158-b33f-56a72c5a92a0" containerName="manager" Jul 18 00:55:22.829698 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:22.829631 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9d98bf0-2c41-4158-b33f-56a72c5a92a0" containerName="manager" Jul 18 00:55:22.832681 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:22.832652 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b7ff4fd55-nv7nj" Jul 18 00:55:22.836140 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:22.836108 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4mw4g\"" Jul 18 00:55:22.836140 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:22.836127 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 18 00:55:22.836140 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:22.836108 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 18 00:55:22.841600 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:22.841575 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b7ff4fd55-nv7nj"] Jul 18 00:55:22.971341 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:22.971307 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msnck\" (UniqueName: \"kubernetes.io/projected/b3992067-b383-4b24-8ae2-a3c230b8bdc7-kube-api-access-msnck\") pod \"maas-controller-7b7ff4fd55-nv7nj\" (UID: \"b3992067-b383-4b24-8ae2-a3c230b8bdc7\") " pod="opendatahub/maas-controller-7b7ff4fd55-nv7nj" Jul 18 00:55:22.971518 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:22.971371 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3992067-b383-4b24-8ae2-a3c230b8bdc7-webhook-cert\") pod \"maas-controller-7b7ff4fd55-nv7nj\" (UID: \"b3992067-b383-4b24-8ae2-a3c230b8bdc7\") " pod="opendatahub/maas-controller-7b7ff4fd55-nv7nj" Jul 18 00:55:23.072859 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:23.072818 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3992067-b383-4b24-8ae2-a3c230b8bdc7-webhook-cert\") pod \"maas-controller-7b7ff4fd55-nv7nj\" (UID: \"b3992067-b383-4b24-8ae2-a3c230b8bdc7\") " pod="opendatahub/maas-controller-7b7ff4fd55-nv7nj" Jul 18 00:55:23.073033 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:23.072897 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msnck\" (UniqueName: \"kubernetes.io/projected/b3992067-b383-4b24-8ae2-a3c230b8bdc7-kube-api-access-msnck\") pod \"maas-controller-7b7ff4fd55-nv7nj\" (UID: \"b3992067-b383-4b24-8ae2-a3c230b8bdc7\") " pod="opendatahub/maas-controller-7b7ff4fd55-nv7nj" Jul 18 00:55:23.075467 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:23.075448 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3992067-b383-4b24-8ae2-a3c230b8bdc7-webhook-cert\") pod \"maas-controller-7b7ff4fd55-nv7nj\" (UID: \"b3992067-b383-4b24-8ae2-a3c230b8bdc7\") " pod="opendatahub/maas-controller-7b7ff4fd55-nv7nj" Jul 18 00:55:23.097223 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:23.097143 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msnck\" (UniqueName: \"kubernetes.io/projected/b3992067-b383-4b24-8ae2-a3c230b8bdc7-kube-api-access-msnck\") pod \"maas-controller-7b7ff4fd55-nv7nj\" (UID: \"b3992067-b383-4b24-8ae2-a3c230b8bdc7\") " pod="opendatahub/maas-controller-7b7ff4fd55-nv7nj" Jul 18 00:55:23.165122 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:23.165081 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b7ff4fd55-nv7nj" Jul 18 00:55:23.300695 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:23.300652 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b7ff4fd55-nv7nj"] Jul 18 00:55:23.302167 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:55:23.302143 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3992067_b383_4b24_8ae2_a3c230b8bdc7.slice/crio-803d8fd7b82d98be59f679bc66b9d6c70db472c9cc8b45b96ee06870f587286b WatchSource:0}: Error finding container 803d8fd7b82d98be59f679bc66b9d6c70db472c9cc8b45b96ee06870f587286b: Status 404 returned error can't find the container with id 803d8fd7b82d98be59f679bc66b9d6c70db472c9cc8b45b96ee06870f587286b Jul 18 00:55:23.335744 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:23.335701 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b7ff4fd55-nv7nj" event={"ID":"b3992067-b383-4b24-8ae2-a3c230b8bdc7","Type":"ContainerStarted","Data":"803d8fd7b82d98be59f679bc66b9d6c70db472c9cc8b45b96ee06870f587286b"} Jul 18 00:55:24.345882 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:24.345843 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b7ff4fd55-nv7nj" event={"ID":"b3992067-b383-4b24-8ae2-a3c230b8bdc7","Type":"ContainerStarted","Data":"510532506a660108af2de251f79f8949f823f243a982b4740301388e77669074"} Jul 18 00:55:24.346246 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:24.345971 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b7ff4fd55-nv7nj" Jul 18 00:55:24.365972 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:24.365908 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7b7ff4fd55-nv7nj" podStartSLOduration=1.8985076429999999 podStartE2EDuration="2.365891729s" totalImagesPullingTime="467.384086ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:55:24.364023853 +0000 UTC m=+884.715874198" watchObservedRunningTime="2026-07-18 00:55:24.365891729 +0000 UTC m=+884.717742073" Jul 18 00:55:35.358920 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:35.358842 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b7ff4fd55-nv7nj" Jul 18 00:55:39.994050 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:39.994012 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b7ff4fd55-nv7nj"] Jul 18 00:55:39.994463 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:39.994272 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7b7ff4fd55-nv7nj" podUID="b3992067-b383-4b24-8ae2-a3c230b8bdc7" containerName="manager" containerID="cri-o://510532506a660108af2de251f79f8949f823f243a982b4740301388e77669074" gracePeriod=30 Jul 18 00:55:40.432724 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:40.432688 2580 generic.go:356] "Generic (PLEG): container finished" podID="b3992067-b383-4b24-8ae2-a3c230b8bdc7" containerID="510532506a660108af2de251f79f8949f823f243a982b4740301388e77669074" exitCode=0 Jul 18 00:55:40.432724 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:40.432709 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b7ff4fd55-nv7nj" event={"ID":"b3992067-b383-4b24-8ae2-a3c230b8bdc7","Type":"ContainerDied","Data":"510532506a660108af2de251f79f8949f823f243a982b4740301388e77669074"} Jul 18 00:55:40.839704 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:40.839655 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b7ff4fd55-nv7nj" Jul 18 00:55:40.926653 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:40.926618 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msnck\" (UniqueName: \"kubernetes.io/projected/b3992067-b383-4b24-8ae2-a3c230b8bdc7-kube-api-access-msnck\") pod \"b3992067-b383-4b24-8ae2-a3c230b8bdc7\" (UID: \"b3992067-b383-4b24-8ae2-a3c230b8bdc7\") " Jul 18 00:55:40.926876 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:40.926723 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3992067-b383-4b24-8ae2-a3c230b8bdc7-webhook-cert\") pod \"b3992067-b383-4b24-8ae2-a3c230b8bdc7\" (UID: \"b3992067-b383-4b24-8ae2-a3c230b8bdc7\") " Jul 18 00:55:40.928870 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:40.928844 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3992067-b383-4b24-8ae2-a3c230b8bdc7-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b3992067-b383-4b24-8ae2-a3c230b8bdc7" (UID: "b3992067-b383-4b24-8ae2-a3c230b8bdc7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 00:55:40.928984 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:40.928842 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3992067-b383-4b24-8ae2-a3c230b8bdc7-kube-api-access-msnck" (OuterVolumeSpecName: "kube-api-access-msnck") pod "b3992067-b383-4b24-8ae2-a3c230b8bdc7" (UID: "b3992067-b383-4b24-8ae2-a3c230b8bdc7"). InnerVolumeSpecName "kube-api-access-msnck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 00:55:41.027523 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:41.027481 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msnck\" (UniqueName: \"kubernetes.io/projected/b3992067-b383-4b24-8ae2-a3c230b8bdc7-kube-api-access-msnck\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:55:41.027523 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:41.027519 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3992067-b383-4b24-8ae2-a3c230b8bdc7-webhook-cert\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 00:55:41.439535 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:41.439461 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b7ff4fd55-nv7nj" Jul 18 00:55:41.439535 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:41.439498 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b7ff4fd55-nv7nj" event={"ID":"b3992067-b383-4b24-8ae2-a3c230b8bdc7","Type":"ContainerDied","Data":"803d8fd7b82d98be59f679bc66b9d6c70db472c9cc8b45b96ee06870f587286b"} Jul 18 00:55:41.439734 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:41.439562 2580 scope.go:117] "RemoveContainer" containerID="510532506a660108af2de251f79f8949f823f243a982b4740301388e77669074" Jul 18 00:55:41.463432 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:41.463396 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b7ff4fd55-nv7nj"] Jul 18 00:55:41.465493 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:41.465470 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7b7ff4fd55-nv7nj"] Jul 18 00:55:42.150135 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:42.150103 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3992067-b383-4b24-8ae2-a3c230b8bdc7" path="/var/lib/kubelet/pods/b3992067-b383-4b24-8ae2-a3c230b8bdc7/volumes" Jul 18 00:55:42.216099 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:42.216055 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b7ff4fd55-2ltkx"] Jul 18 00:55:42.216594 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:42.216563 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3992067-b383-4b24-8ae2-a3c230b8bdc7" containerName="manager" Jul 18 00:55:42.216594 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:42.216589 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3992067-b383-4b24-8ae2-a3c230b8bdc7" containerName="manager" Jul 18 00:55:42.216784 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:42.216709 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3992067-b383-4b24-8ae2-a3c230b8bdc7" containerName="manager" Jul 18 00:55:42.219837 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:42.219819 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b7ff4fd55-2ltkx" Jul 18 00:55:42.223335 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:42.223308 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 18 00:55:42.223458 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:42.223342 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 18 00:55:42.223458 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:42.223369 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4mw4g\"" Jul 18 00:55:42.229655 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:42.229628 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b7ff4fd55-2ltkx"] Jul 18 00:55:42.234950 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:42.234919 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/04dd6278-f1d4-4b48-8026-de1958a1123b-webhook-cert\") pod \"maas-controller-7b7ff4fd55-2ltkx\" (UID: \"04dd6278-f1d4-4b48-8026-de1958a1123b\") " pod="opendatahub/maas-controller-7b7ff4fd55-2ltkx" Jul 18 00:55:42.235080 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:42.234997 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d7gm\" (UniqueName: \"kubernetes.io/projected/04dd6278-f1d4-4b48-8026-de1958a1123b-kube-api-access-7d7gm\") pod \"maas-controller-7b7ff4fd55-2ltkx\" (UID: \"04dd6278-f1d4-4b48-8026-de1958a1123b\") " pod="opendatahub/maas-controller-7b7ff4fd55-2ltkx" Jul 18 00:55:42.335922 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:42.335885 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/04dd6278-f1d4-4b48-8026-de1958a1123b-webhook-cert\") pod \"maas-controller-7b7ff4fd55-2ltkx\" (UID: \"04dd6278-f1d4-4b48-8026-de1958a1123b\") " pod="opendatahub/maas-controller-7b7ff4fd55-2ltkx" Jul 18 00:55:42.336097 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:42.335943 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d7gm\" (UniqueName: \"kubernetes.io/projected/04dd6278-f1d4-4b48-8026-de1958a1123b-kube-api-access-7d7gm\") pod \"maas-controller-7b7ff4fd55-2ltkx\" (UID: \"04dd6278-f1d4-4b48-8026-de1958a1123b\") " pod="opendatahub/maas-controller-7b7ff4fd55-2ltkx" Jul 18 00:55:42.338470 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:42.338451 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/04dd6278-f1d4-4b48-8026-de1958a1123b-webhook-cert\") pod \"maas-controller-7b7ff4fd55-2ltkx\" (UID: \"04dd6278-f1d4-4b48-8026-de1958a1123b\") " pod="opendatahub/maas-controller-7b7ff4fd55-2ltkx" Jul 18 00:55:42.345566 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:42.345536 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d7gm\" (UniqueName: \"kubernetes.io/projected/04dd6278-f1d4-4b48-8026-de1958a1123b-kube-api-access-7d7gm\") pod \"maas-controller-7b7ff4fd55-2ltkx\" (UID: \"04dd6278-f1d4-4b48-8026-de1958a1123b\") " pod="opendatahub/maas-controller-7b7ff4fd55-2ltkx" Jul 18 00:55:42.530866 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:42.530830 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b7ff4fd55-2ltkx" Jul 18 00:55:42.668529 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:42.668505 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b7ff4fd55-2ltkx"] Jul 18 00:55:42.670415 ip-10-0-134-239 kubenswrapper[2580]: W0718 00:55:42.670388 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04dd6278_f1d4_4b48_8026_de1958a1123b.slice/crio-4f3aa8c10f890fe58036b96c226ffb45ddead2b8de5166367bf40acfd630d509 WatchSource:0}: Error finding container 4f3aa8c10f890fe58036b96c226ffb45ddead2b8de5166367bf40acfd630d509: Status 404 returned error can't find the container with id 4f3aa8c10f890fe58036b96c226ffb45ddead2b8de5166367bf40acfd630d509 Jul 18 00:55:43.452011 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:43.451920 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b7ff4fd55-2ltkx" event={"ID":"04dd6278-f1d4-4b48-8026-de1958a1123b","Type":"ContainerStarted","Data":"5225c668df12b62234753cade7ed345c4b22e41b83c7066544768f2c7d15c357"} Jul 18 00:55:43.452011 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:43.451969 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b7ff4fd55-2ltkx" event={"ID":"04dd6278-f1d4-4b48-8026-de1958a1123b","Type":"ContainerStarted","Data":"4f3aa8c10f890fe58036b96c226ffb45ddead2b8de5166367bf40acfd630d509"} Jul 18 00:55:43.452415 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:43.452041 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b7ff4fd55-2ltkx" Jul 18 00:55:43.468875 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:43.468827 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7b7ff4fd55-2ltkx" podStartSLOduration=1.122504917 podStartE2EDuration="1.468809826s" totalImagesPullingTime="346.304909ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 00:55:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 00:55:43.467257995 +0000 UTC m=+903.819108339" watchObservedRunningTime="2026-07-18 00:55:43.468809826 +0000 UTC m=+903.820660196" Jul 18 00:55:54.460185 ip-10-0-134-239 kubenswrapper[2580]: I0718 00:55:54.460152 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b7ff4fd55-2ltkx" Jul 18 01:07:47.886593 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:07:47.886554 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz"] Jul 18 01:07:47.889949 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:07:47.889927 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" Jul 18 01:07:47.893170 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:07:47.893150 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2djvc\"" Jul 18 01:07:47.893294 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:07:47.893193 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-354f15cb-serving-cert\"" Jul 18 01:07:47.919959 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:07:47.919929 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4ncp\" (UniqueName: \"kubernetes.io/projected/2f2e700e-b1a5-421b-8bac-c15e729b0e9b-kube-api-access-s4ncp\") pod \"maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz\" (UID: \"2f2e700e-b1a5-421b-8bac-c15e729b0e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" Jul 18 01:07:47.920086 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:07:47.919968 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f2e700e-b1a5-421b-8bac-c15e729b0e9b-openshift-service-ca\") pod \"maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz\" (UID: \"2f2e700e-b1a5-421b-8bac-c15e729b0e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" Jul 18 01:07:47.920086 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:07:47.920025 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f2e700e-b1a5-421b-8bac-c15e729b0e9b-maas-api-tls\") pod \"maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz\" (UID: \"2f2e700e-b1a5-421b-8bac-c15e729b0e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" Jul 18 01:07:47.924394 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:07:47.924366 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz"] Jul 18 01:07:48.020532 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:07:48.020499 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4ncp\" (UniqueName: \"kubernetes.io/projected/2f2e700e-b1a5-421b-8bac-c15e729b0e9b-kube-api-access-s4ncp\") pod \"maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz\" (UID: \"2f2e700e-b1a5-421b-8bac-c15e729b0e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" Jul 18 01:07:48.020532 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:07:48.020537 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f2e700e-b1a5-421b-8bac-c15e729b0e9b-openshift-service-ca\") pod \"maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz\" (UID: \"2f2e700e-b1a5-421b-8bac-c15e729b0e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" Jul 18 01:07:48.020800 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:07:48.020569 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f2e700e-b1a5-421b-8bac-c15e729b0e9b-maas-api-tls\") pod \"maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz\" (UID: \"2f2e700e-b1a5-421b-8bac-c15e729b0e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" Jul 18 01:07:48.020800 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:07:48.020711 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-serving-cert: secret "maas-api-e2e-ait-354f15cb-serving-cert" not found Jul 18 01:07:48.020800 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:07:48.020764 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f2e700e-b1a5-421b-8bac-c15e729b0e9b-maas-api-tls podName:2f2e700e-b1a5-421b-8bac-c15e729b0e9b nodeName:}" failed. No retries permitted until 2026-07-18 01:07:48.520750025 +0000 UTC m=+1628.872600348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2f2e700e-b1a5-421b-8bac-c15e729b0e9b-maas-api-tls") pod "maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" (UID: "2f2e700e-b1a5-421b-8bac-c15e729b0e9b") : secret "maas-api-e2e-ait-354f15cb-serving-cert" not found Jul 18 01:07:48.021413 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:07:48.021390 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f2e700e-b1a5-421b-8bac-c15e729b0e9b-openshift-service-ca\") pod \"maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz\" (UID: \"2f2e700e-b1a5-421b-8bac-c15e729b0e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" Jul 18 01:07:48.030659 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:07:48.030631 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4ncp\" (UniqueName: \"kubernetes.io/projected/2f2e700e-b1a5-421b-8bac-c15e729b0e9b-kube-api-access-s4ncp\") pod \"maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz\" (UID: \"2f2e700e-b1a5-421b-8bac-c15e729b0e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" Jul 18 01:07:48.524881 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:07:48.524841 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f2e700e-b1a5-421b-8bac-c15e729b0e9b-maas-api-tls\") pod \"maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz\" (UID: \"2f2e700e-b1a5-421b-8bac-c15e729b0e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" Jul 18 01:07:48.527547 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:07:48.527522 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f2e700e-b1a5-421b-8bac-c15e729b0e9b-maas-api-tls\") pod \"maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz\" (UID: \"2f2e700e-b1a5-421b-8bac-c15e729b0e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" Jul 18 01:07:48.800194 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:07:48.800119 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" Jul 18 01:07:48.936537 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:07:48.936509 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz"] Jul 18 01:07:48.938403 ip-10-0-134-239 kubenswrapper[2580]: W0718 01:07:48.938359 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f2e700e_b1a5_421b_8bac_c15e729b0e9b.slice/crio-aa4264bc925ef9658145903721f3f728c9114890d061935329647590d452845d WatchSource:0}: Error finding container aa4264bc925ef9658145903721f3f728c9114890d061935329647590d452845d: Status 404 returned error can't find the container with id aa4264bc925ef9658145903721f3f728c9114890d061935329647590d452845d Jul 18 01:07:48.939876 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:07:48.939856 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 01:07:49.069063 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:07:49.068980 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" event={"ID":"2f2e700e-b1a5-421b-8bac-c15e729b0e9b","Type":"ContainerStarted","Data":"aa4264bc925ef9658145903721f3f728c9114890d061935329647590d452845d"} Jul 18 01:07:50.075372 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:07:50.075339 2580 generic.go:356] "Generic (PLEG): container finished" podID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" containerID="eeaaccf1f5c4c819e2052ad640d9cdc635e51bc318a11488fb9900797bc736f6" exitCode=1 Jul 18 01:07:50.075809 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:07:50.075434 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" event={"ID":"2f2e700e-b1a5-421b-8bac-c15e729b0e9b","Type":"ContainerDied","Data":"eeaaccf1f5c4c819e2052ad640d9cdc635e51bc318a11488fb9900797bc736f6"} Jul 18 01:07:50.075809 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:07:50.075708 2580 scope.go:117] "RemoveContainer" containerID="eeaaccf1f5c4c819e2052ad640d9cdc635e51bc318a11488fb9900797bc736f6" Jul 18 01:07:51.082225 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:07:51.082141 2580 generic.go:356] "Generic (PLEG): container finished" podID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" containerID="e47d3a468918239a19cbc1ec97b84023e43f0739598d44266f17fdc9ef5c2f34" exitCode=1 Jul 18 01:07:51.082694 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:07:51.082227 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" event={"ID":"2f2e700e-b1a5-421b-8bac-c15e729b0e9b","Type":"ContainerDied","Data":"e47d3a468918239a19cbc1ec97b84023e43f0739598d44266f17fdc9ef5c2f34"} Jul 18 01:07:51.082694 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:07:51.082283 2580 scope.go:117] "RemoveContainer" containerID="eeaaccf1f5c4c819e2052ad640d9cdc635e51bc318a11488fb9900797bc736f6" Jul 18 01:07:51.082694 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:07:51.082441 2580 scope.go:117] "RemoveContainer" containerID="e47d3a468918239a19cbc1ec97b84023e43f0739598d44266f17fdc9ef5c2f34" Jul 18 01:07:51.082862 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:07:51.082772 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz_odh-ai-gateway-infra(2f2e700e-b1a5-421b-8bac-c15e729b0e9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" Jul 18 01:07:52.088991 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:07:52.088965 2580 scope.go:117] "RemoveContainer" containerID="e47d3a468918239a19cbc1ec97b84023e43f0739598d44266f17fdc9ef5c2f34" Jul 18 01:07:52.089375 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:07:52.089152 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz_odh-ai-gateway-infra(2f2e700e-b1a5-421b-8bac-c15e729b0e9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" Jul 18 01:07:53.801145 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:07:53.801109 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" Jul 18 01:07:53.801531 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:07:53.801509 2580 scope.go:117] "RemoveContainer" containerID="e47d3a468918239a19cbc1ec97b84023e43f0739598d44266f17fdc9ef5c2f34" Jul 18 01:07:53.801730 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:07:53.801706 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz_odh-ai-gateway-infra(2f2e700e-b1a5-421b-8bac-c15e729b0e9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" Jul 18 01:07:58.800690 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:07:58.800649 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" Jul 18 01:07:58.801071 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:07:58.801058 2580 scope.go:117] "RemoveContainer" containerID="e47d3a468918239a19cbc1ec97b84023e43f0739598d44266f17fdc9ef5c2f34" Jul 18 01:07:58.801251 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:07:58.801232 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz_odh-ai-gateway-infra(2f2e700e-b1a5-421b-8bac-c15e729b0e9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" Jul 18 01:08:14.143883 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:08:14.143848 2580 scope.go:117] "RemoveContainer" containerID="e47d3a468918239a19cbc1ec97b84023e43f0739598d44266f17fdc9ef5c2f34" Jul 18 01:08:15.208895 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:08:15.208806 2580 generic.go:356] "Generic (PLEG): container finished" podID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" containerID="358bd689e71987842071cd5fa19c39dfc32f08a135cd348ca946b25c6125a3e5" exitCode=1 Jul 18 01:08:15.208895 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:08:15.208858 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" event={"ID":"2f2e700e-b1a5-421b-8bac-c15e729b0e9b","Type":"ContainerDied","Data":"358bd689e71987842071cd5fa19c39dfc32f08a135cd348ca946b25c6125a3e5"} Jul 18 01:08:15.209301 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:08:15.208954 2580 scope.go:117] "RemoveContainer" containerID="e47d3a468918239a19cbc1ec97b84023e43f0739598d44266f17fdc9ef5c2f34" Jul 18 01:08:15.209364 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:08:15.209344 2580 scope.go:117] "RemoveContainer" containerID="358bd689e71987842071cd5fa19c39dfc32f08a135cd348ca946b25c6125a3e5" Jul 18 01:08:15.209594 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:08:15.209570 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz_odh-ai-gateway-infra(2f2e700e-b1a5-421b-8bac-c15e729b0e9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" Jul 18 01:08:18.800245 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:08:18.800210 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" Jul 18 01:08:18.800837 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:08:18.800262 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" Jul 18 01:08:18.800837 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:08:18.800569 2580 scope.go:117] "RemoveContainer" containerID="358bd689e71987842071cd5fa19c39dfc32f08a135cd348ca946b25c6125a3e5" Jul 18 01:08:18.800837 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:08:18.800759 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz_odh-ai-gateway-infra(2f2e700e-b1a5-421b-8bac-c15e729b0e9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" Jul 18 01:08:33.144087 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:08:33.144051 2580 scope.go:117] "RemoveContainer" containerID="358bd689e71987842071cd5fa19c39dfc32f08a135cd348ca946b25c6125a3e5" Jul 18 01:08:33.144474 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:08:33.144225 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz_odh-ai-gateway-infra(2f2e700e-b1a5-421b-8bac-c15e729b0e9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" Jul 18 01:08:47.143322 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:08:47.143289 2580 scope.go:117] "RemoveContainer" containerID="358bd689e71987842071cd5fa19c39dfc32f08a135cd348ca946b25c6125a3e5" Jul 18 01:08:48.384455 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:08:48.384420 2580 generic.go:356] "Generic (PLEG): container finished" podID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" containerID="37bee5be9eb1d5622fb9cb2b7df77091b1bd51d2478b90609cae66e0584a5d52" exitCode=1 Jul 18 01:08:48.384880 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:08:48.384494 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" event={"ID":"2f2e700e-b1a5-421b-8bac-c15e729b0e9b","Type":"ContainerDied","Data":"37bee5be9eb1d5622fb9cb2b7df77091b1bd51d2478b90609cae66e0584a5d52"} Jul 18 01:08:48.384880 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:08:48.384559 2580 scope.go:117] "RemoveContainer" containerID="358bd689e71987842071cd5fa19c39dfc32f08a135cd348ca946b25c6125a3e5" Jul 18 01:08:48.384977 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:08:48.384957 2580 scope.go:117] "RemoveContainer" containerID="37bee5be9eb1d5622fb9cb2b7df77091b1bd51d2478b90609cae66e0584a5d52" Jul 18 01:08:48.385171 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:08:48.385144 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz_odh-ai-gateway-infra(2f2e700e-b1a5-421b-8bac-c15e729b0e9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" Jul 18 01:08:48.801205 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:08:48.801175 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" Jul 18 01:08:48.801388 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:08:48.801227 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" Jul 18 01:08:49.391837 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:08:49.391810 2580 scope.go:117] "RemoveContainer" containerID="37bee5be9eb1d5622fb9cb2b7df77091b1bd51d2478b90609cae66e0584a5d52" Jul 18 01:08:49.392235 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:08:49.391985 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz_odh-ai-gateway-infra(2f2e700e-b1a5-421b-8bac-c15e729b0e9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" Jul 18 01:09:03.143590 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:09:03.143510 2580 scope.go:117] "RemoveContainer" containerID="37bee5be9eb1d5622fb9cb2b7df77091b1bd51d2478b90609cae66e0584a5d52" Jul 18 01:09:03.144017 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:09:03.143690 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz_odh-ai-gateway-infra(2f2e700e-b1a5-421b-8bac-c15e729b0e9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" Jul 18 01:09:14.149792 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:09:14.149760 2580 scope.go:117] "RemoveContainer" containerID="37bee5be9eb1d5622fb9cb2b7df77091b1bd51d2478b90609cae66e0584a5d52" Jul 18 01:09:14.150206 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:09:14.149966 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz_odh-ai-gateway-infra(2f2e700e-b1a5-421b-8bac-c15e729b0e9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" Jul 18 01:09:27.143428 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:09:27.143399 2580 scope.go:117] "RemoveContainer" containerID="37bee5be9eb1d5622fb9cb2b7df77091b1bd51d2478b90609cae66e0584a5d52" Jul 18 01:09:27.143841 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:09:27.143570 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz_odh-ai-gateway-infra(2f2e700e-b1a5-421b-8bac-c15e729b0e9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" Jul 18 01:09:38.143616 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:09:38.143583 2580 scope.go:117] "RemoveContainer" containerID="37bee5be9eb1d5622fb9cb2b7df77091b1bd51d2478b90609cae66e0584a5d52" Jul 18 01:09:38.637815 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:09:38.637770 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" event={"ID":"2f2e700e-b1a5-421b-8bac-c15e729b0e9b","Type":"ContainerStarted","Data":"5ec5c3b2496f76025f70545ba3ef17ce949cea6247734006cc692ad49e0f6d16"} Jul 18 01:09:38.638005 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:09:38.637990 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" Jul 18 01:09:38.654573 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:09:38.654522 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" podStartSLOduration=109.698270978 podStartE2EDuration="1m51.654505119s" totalImagesPullingTime="1.956234141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:07:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:09:38.653352402 +0000 UTC m=+1739.005202742" watchObservedRunningTime="2026-07-18 01:09:38.654505119 +0000 UTC m=+1739.006355463" Jul 18 01:09:39.644960 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:09:39.644925 2580 generic.go:356] "Generic (PLEG): container finished" podID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" containerID="5ec5c3b2496f76025f70545ba3ef17ce949cea6247734006cc692ad49e0f6d16" exitCode=1 Jul 18 01:09:39.645350 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:09:39.645002 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" event={"ID":"2f2e700e-b1a5-421b-8bac-c15e729b0e9b","Type":"ContainerDied","Data":"5ec5c3b2496f76025f70545ba3ef17ce949cea6247734006cc692ad49e0f6d16"} Jul 18 01:09:39.645350 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:09:39.645068 2580 scope.go:117] "RemoveContainer" containerID="37bee5be9eb1d5622fb9cb2b7df77091b1bd51d2478b90609cae66e0584a5d52" Jul 18 01:09:39.645350 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:09:39.645218 2580 scope.go:117] "RemoveContainer" containerID="5ec5c3b2496f76025f70545ba3ef17ce949cea6247734006cc692ad49e0f6d16" Jul 18 01:09:39.645486 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:09:39.645465 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz_odh-ai-gateway-infra(2f2e700e-b1a5-421b-8bac-c15e729b0e9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" Jul 18 01:09:40.651607 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:09:40.651575 2580 scope.go:117] "RemoveContainer" containerID="5ec5c3b2496f76025f70545ba3ef17ce949cea6247734006cc692ad49e0f6d16" Jul 18 01:09:40.652017 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:09:40.651787 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz_odh-ai-gateway-infra(2f2e700e-b1a5-421b-8bac-c15e729b0e9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" Jul 18 01:09:48.800428 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:09:48.800392 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" Jul 18 01:09:48.800914 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:09:48.800807 2580 scope.go:117] "RemoveContainer" containerID="5ec5c3b2496f76025f70545ba3ef17ce949cea6247734006cc692ad49e0f6d16" Jul 18 01:09:48.800996 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:09:48.800977 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz_odh-ai-gateway-infra(2f2e700e-b1a5-421b-8bac-c15e729b0e9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" Jul 18 01:10:04.143536 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:10:04.143497 2580 scope.go:117] "RemoveContainer" containerID="5ec5c3b2496f76025f70545ba3ef17ce949cea6247734006cc692ad49e0f6d16" Jul 18 01:10:04.144180 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:10:04.143712 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz_odh-ai-gateway-infra(2f2e700e-b1a5-421b-8bac-c15e729b0e9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" Jul 18 01:10:18.144372 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:10:18.144338 2580 scope.go:117] "RemoveContainer" containerID="5ec5c3b2496f76025f70545ba3ef17ce949cea6247734006cc692ad49e0f6d16" Jul 18 01:10:18.144795 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:10:18.144509 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz_odh-ai-gateway-infra(2f2e700e-b1a5-421b-8bac-c15e729b0e9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" Jul 18 01:10:31.143972 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:10:31.143893 2580 scope.go:117] "RemoveContainer" containerID="5ec5c3b2496f76025f70545ba3ef17ce949cea6247734006cc692ad49e0f6d16" Jul 18 01:10:31.144328 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:10:31.144054 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz_odh-ai-gateway-infra(2f2e700e-b1a5-421b-8bac-c15e729b0e9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" Jul 18 01:10:43.144217 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:10:43.144179 2580 scope.go:117] "RemoveContainer" containerID="5ec5c3b2496f76025f70545ba3ef17ce949cea6247734006cc692ad49e0f6d16" Jul 18 01:10:43.144656 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:10:43.144371 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz_odh-ai-gateway-infra(2f2e700e-b1a5-421b-8bac-c15e729b0e9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" Jul 18 01:10:57.143605 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:10:57.143571 2580 scope.go:117] "RemoveContainer" containerID="5ec5c3b2496f76025f70545ba3ef17ce949cea6247734006cc692ad49e0f6d16" Jul 18 01:10:57.144030 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:10:57.143791 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz_odh-ai-gateway-infra(2f2e700e-b1a5-421b-8bac-c15e729b0e9b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" Jul 18 01:10:58.289252 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:10:58.289203 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz"] Jul 18 01:10:58.430534 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:10:58.430467 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" Jul 18 01:10:58.469307 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:10:58.469273 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f2e700e-b1a5-421b-8bac-c15e729b0e9b-maas-api-tls\") pod \"2f2e700e-b1a5-421b-8bac-c15e729b0e9b\" (UID: \"2f2e700e-b1a5-421b-8bac-c15e729b0e9b\") " Jul 18 01:10:58.469493 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:10:58.469314 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4ncp\" (UniqueName: \"kubernetes.io/projected/2f2e700e-b1a5-421b-8bac-c15e729b0e9b-kube-api-access-s4ncp\") pod \"2f2e700e-b1a5-421b-8bac-c15e729b0e9b\" (UID: \"2f2e700e-b1a5-421b-8bac-c15e729b0e9b\") " Jul 18 01:10:58.469493 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:10:58.469369 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f2e700e-b1a5-421b-8bac-c15e729b0e9b-openshift-service-ca\") pod \"2f2e700e-b1a5-421b-8bac-c15e729b0e9b\" (UID: \"2f2e700e-b1a5-421b-8bac-c15e729b0e9b\") " Jul 18 01:10:58.469847 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:10:58.469822 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f2e700e-b1a5-421b-8bac-c15e729b0e9b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2f2e700e-b1a5-421b-8bac-c15e729b0e9b" (UID: "2f2e700e-b1a5-421b-8bac-c15e729b0e9b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 01:10:58.471814 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:10:58.471790 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f2e700e-b1a5-421b-8bac-c15e729b0e9b-kube-api-access-s4ncp" (OuterVolumeSpecName: "kube-api-access-s4ncp") pod "2f2e700e-b1a5-421b-8bac-c15e729b0e9b" (UID: "2f2e700e-b1a5-421b-8bac-c15e729b0e9b"). InnerVolumeSpecName "kube-api-access-s4ncp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 01:10:58.471814 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:10:58.471796 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f2e700e-b1a5-421b-8bac-c15e729b0e9b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2f2e700e-b1a5-421b-8bac-c15e729b0e9b" (UID: "2f2e700e-b1a5-421b-8bac-c15e729b0e9b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 01:10:58.570473 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:10:58.570434 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f2e700e-b1a5-421b-8bac-c15e729b0e9b-openshift-service-ca\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:10:58.570473 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:10:58.570469 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f2e700e-b1a5-421b-8bac-c15e729b0e9b-maas-api-tls\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:10:58.570473 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:10:58.570478 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4ncp\" (UniqueName: \"kubernetes.io/projected/2f2e700e-b1a5-421b-8bac-c15e729b0e9b-kube-api-access-s4ncp\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:10:59.047870 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:10:59.047835 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" Jul 18 01:10:59.047870 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:10:59.047848 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz" event={"ID":"2f2e700e-b1a5-421b-8bac-c15e729b0e9b","Type":"ContainerDied","Data":"aa4264bc925ef9658145903721f3f728c9114890d061935329647590d452845d"} Jul 18 01:10:59.048082 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:10:59.047940 2580 scope.go:117] "RemoveContainer" containerID="5ec5c3b2496f76025f70545ba3ef17ce949cea6247734006cc692ad49e0f6d16" Jul 18 01:10:59.068746 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:10:59.068720 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz"] Jul 18 01:10:59.072073 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:10:59.072046 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-354f15cb-7cd5fc65b9-ckrfz"] Jul 18 01:11:00.149245 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:00.149211 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" path="/var/lib/kubelet/pods/2f2e700e-b1a5-421b-8bac-c15e729b0e9b/volumes" Jul 18 01:11:06.025004 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:06.024968 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z"] Jul 18 01:11:06.025513 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:06.025492 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" containerName="maas-api" Jul 18 01:11:06.025592 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:06.025517 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" containerName="maas-api" Jul 18 01:11:06.025592 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:06.025532 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" containerName="maas-api" Jul 18 01:11:06.025592 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:06.025540 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" containerName="maas-api" Jul 18 01:11:06.025592 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:06.025553 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" containerName="maas-api" Jul 18 01:11:06.025592 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:06.025561 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" containerName="maas-api" Jul 18 01:11:06.025592 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:06.025590 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" containerName="maas-api" Jul 18 01:11:06.025886 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:06.025598 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" containerName="maas-api" Jul 18 01:11:06.025886 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:06.025706 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" containerName="maas-api" Jul 18 01:11:06.025886 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:06.025719 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" containerName="maas-api" Jul 18 01:11:06.025886 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:06.025729 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" containerName="maas-api" Jul 18 01:11:06.030505 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:06.030480 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" Jul 18 01:11:06.033095 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:06.033071 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-1b6716ba-serving-cert\"" Jul 18 01:11:06.033216 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:06.033113 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2djvc\"" Jul 18 01:11:06.038598 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:06.038187 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z"] Jul 18 01:11:06.139870 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:06.139831 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/532fb4f1-aa6a-4196-84de-2856b99cf0ec-maas-api-tls\") pod \"maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z\" (UID: \"532fb4f1-aa6a-4196-84de-2856b99cf0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" Jul 18 01:11:06.139870 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:06.139873 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/532fb4f1-aa6a-4196-84de-2856b99cf0ec-openshift-service-ca\") pod \"maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z\" (UID: \"532fb4f1-aa6a-4196-84de-2856b99cf0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" Jul 18 01:11:06.140100 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:06.139939 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxnrw\" (UniqueName: \"kubernetes.io/projected/532fb4f1-aa6a-4196-84de-2856b99cf0ec-kube-api-access-lxnrw\") pod \"maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z\" (UID: \"532fb4f1-aa6a-4196-84de-2856b99cf0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" Jul 18 01:11:06.240580 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:06.240542 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/532fb4f1-aa6a-4196-84de-2856b99cf0ec-maas-api-tls\") pod \"maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z\" (UID: \"532fb4f1-aa6a-4196-84de-2856b99cf0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" Jul 18 01:11:06.240580 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:06.240578 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/532fb4f1-aa6a-4196-84de-2856b99cf0ec-openshift-service-ca\") pod \"maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z\" (UID: \"532fb4f1-aa6a-4196-84de-2856b99cf0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" Jul 18 01:11:06.240817 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:06.240624 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxnrw\" (UniqueName: \"kubernetes.io/projected/532fb4f1-aa6a-4196-84de-2856b99cf0ec-kube-api-access-lxnrw\") pod \"maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z\" (UID: \"532fb4f1-aa6a-4196-84de-2856b99cf0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" Jul 18 01:11:06.240817 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:11:06.240721 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-serving-cert: secret "maas-api-e2e-migrate-1b6716ba-serving-cert" not found Jul 18 01:11:06.240817 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:11:06.240804 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/532fb4f1-aa6a-4196-84de-2856b99cf0ec-maas-api-tls podName:532fb4f1-aa6a-4196-84de-2856b99cf0ec nodeName:}" failed. No retries permitted until 2026-07-18 01:11:06.740786735 +0000 UTC m=+1827.092637059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/532fb4f1-aa6a-4196-84de-2856b99cf0ec-maas-api-tls") pod "maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" (UID: "532fb4f1-aa6a-4196-84de-2856b99cf0ec") : secret "maas-api-e2e-migrate-1b6716ba-serving-cert" not found Jul 18 01:11:06.241442 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:06.241421 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/532fb4f1-aa6a-4196-84de-2856b99cf0ec-openshift-service-ca\") pod \"maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z\" (UID: \"532fb4f1-aa6a-4196-84de-2856b99cf0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" Jul 18 01:11:06.252182 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:06.252149 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxnrw\" (UniqueName: \"kubernetes.io/projected/532fb4f1-aa6a-4196-84de-2856b99cf0ec-kube-api-access-lxnrw\") pod \"maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z\" (UID: \"532fb4f1-aa6a-4196-84de-2856b99cf0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" Jul 18 01:11:06.745538 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:06.745502 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/532fb4f1-aa6a-4196-84de-2856b99cf0ec-maas-api-tls\") pod \"maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z\" (UID: \"532fb4f1-aa6a-4196-84de-2856b99cf0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" Jul 18 01:11:06.748435 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:06.748407 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/532fb4f1-aa6a-4196-84de-2856b99cf0ec-maas-api-tls\") pod \"maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z\" (UID: \"532fb4f1-aa6a-4196-84de-2856b99cf0ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" Jul 18 01:11:06.943252 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:06.943212 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" Jul 18 01:11:07.086583 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:07.086548 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z"] Jul 18 01:11:07.089202 ip-10-0-134-239 kubenswrapper[2580]: W0718 01:11:07.089164 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod532fb4f1_aa6a_4196_84de_2856b99cf0ec.slice/crio-cc978588d9f63f7892fa275f4652cab4e52e5019548c9cf5366fae0dfccde9fd WatchSource:0}: Error finding container cc978588d9f63f7892fa275f4652cab4e52e5019548c9cf5366fae0dfccde9fd: Status 404 returned error can't find the container with id cc978588d9f63f7892fa275f4652cab4e52e5019548c9cf5366fae0dfccde9fd Jul 18 01:11:08.098942 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:08.098910 2580 generic.go:356] "Generic (PLEG): container finished" podID="532fb4f1-aa6a-4196-84de-2856b99cf0ec" containerID="d9bc96fe5a435d864252a65c328db48b73402e80bb5434c619300efa5a0df8e4" exitCode=1 Jul 18 01:11:08.099421 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:08.099009 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" event={"ID":"532fb4f1-aa6a-4196-84de-2856b99cf0ec","Type":"ContainerDied","Data":"d9bc96fe5a435d864252a65c328db48b73402e80bb5434c619300efa5a0df8e4"} Jul 18 01:11:08.099421 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:08.099062 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" event={"ID":"532fb4f1-aa6a-4196-84de-2856b99cf0ec","Type":"ContainerStarted","Data":"cc978588d9f63f7892fa275f4652cab4e52e5019548c9cf5366fae0dfccde9fd"} Jul 18 01:11:08.099421 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:08.099290 2580 scope.go:117] "RemoveContainer" containerID="d9bc96fe5a435d864252a65c328db48b73402e80bb5434c619300efa5a0df8e4" Jul 18 01:11:09.106811 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:09.106713 2580 generic.go:356] "Generic (PLEG): container finished" podID="532fb4f1-aa6a-4196-84de-2856b99cf0ec" containerID="60264bbed7cc12395ef17c54b97136d4ae10210e1c69bea6fb5c168e07acd02d" exitCode=1 Jul 18 01:11:09.106811 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:09.106798 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" event={"ID":"532fb4f1-aa6a-4196-84de-2856b99cf0ec","Type":"ContainerDied","Data":"60264bbed7cc12395ef17c54b97136d4ae10210e1c69bea6fb5c168e07acd02d"} Jul 18 01:11:09.107295 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:09.106854 2580 scope.go:117] "RemoveContainer" containerID="d9bc96fe5a435d864252a65c328db48b73402e80bb5434c619300efa5a0df8e4" Jul 18 01:11:09.107295 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:09.107083 2580 scope.go:117] "RemoveContainer" containerID="60264bbed7cc12395ef17c54b97136d4ae10210e1c69bea6fb5c168e07acd02d" Jul 18 01:11:09.107295 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:11:09.107282 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z_odh-ai-gateway-infra(532fb4f1-aa6a-4196-84de-2856b99cf0ec)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" podUID="532fb4f1-aa6a-4196-84de-2856b99cf0ec" Jul 18 01:11:10.113940 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:10.113908 2580 scope.go:117] "RemoveContainer" containerID="60264bbed7cc12395ef17c54b97136d4ae10210e1c69bea6fb5c168e07acd02d" Jul 18 01:11:10.114311 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:11:10.114112 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z_odh-ai-gateway-infra(532fb4f1-aa6a-4196-84de-2856b99cf0ec)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" podUID="532fb4f1-aa6a-4196-84de-2856b99cf0ec" Jul 18 01:11:11.943910 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:11.943866 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" Jul 18 01:11:11.944397 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:11.944384 2580 scope.go:117] "RemoveContainer" containerID="60264bbed7cc12395ef17c54b97136d4ae10210e1c69bea6fb5c168e07acd02d" Jul 18 01:11:11.944606 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:11:11.944582 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z_odh-ai-gateway-infra(532fb4f1-aa6a-4196-84de-2856b99cf0ec)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" podUID="532fb4f1-aa6a-4196-84de-2856b99cf0ec" Jul 18 01:11:16.943755 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:16.943713 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" Jul 18 01:11:16.944269 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:16.944250 2580 scope.go:117] "RemoveContainer" containerID="60264bbed7cc12395ef17c54b97136d4ae10210e1c69bea6fb5c168e07acd02d" Jul 18 01:11:16.944511 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:11:16.944486 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z_odh-ai-gateway-infra(532fb4f1-aa6a-4196-84de-2856b99cf0ec)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" podUID="532fb4f1-aa6a-4196-84de-2856b99cf0ec" Jul 18 01:11:32.144281 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:32.144249 2580 scope.go:117] "RemoveContainer" containerID="60264bbed7cc12395ef17c54b97136d4ae10210e1c69bea6fb5c168e07acd02d" Jul 18 01:11:33.244602 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:33.244569 2580 generic.go:356] "Generic (PLEG): container finished" podID="532fb4f1-aa6a-4196-84de-2856b99cf0ec" containerID="0ea3e9f0457149405044a9c3b79a6cf739d0a6d03d612519354a56907ee80502" exitCode=1 Jul 18 01:11:33.245052 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:33.244642 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" event={"ID":"532fb4f1-aa6a-4196-84de-2856b99cf0ec","Type":"ContainerDied","Data":"0ea3e9f0457149405044a9c3b79a6cf739d0a6d03d612519354a56907ee80502"} Jul 18 01:11:33.245052 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:33.244728 2580 scope.go:117] "RemoveContainer" containerID="60264bbed7cc12395ef17c54b97136d4ae10210e1c69bea6fb5c168e07acd02d" Jul 18 01:11:33.245156 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:33.245104 2580 scope.go:117] "RemoveContainer" containerID="0ea3e9f0457149405044a9c3b79a6cf739d0a6d03d612519354a56907ee80502" Jul 18 01:11:33.245328 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:11:33.245305 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z_odh-ai-gateway-infra(532fb4f1-aa6a-4196-84de-2856b99cf0ec)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" podUID="532fb4f1-aa6a-4196-84de-2856b99cf0ec" Jul 18 01:11:36.943459 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:36.943421 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" Jul 18 01:11:36.943890 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:36.943510 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" Jul 18 01:11:36.943890 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:36.943876 2580 scope.go:117] "RemoveContainer" containerID="0ea3e9f0457149405044a9c3b79a6cf739d0a6d03d612519354a56907ee80502" Jul 18 01:11:36.944065 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:11:36.944043 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z_odh-ai-gateway-infra(532fb4f1-aa6a-4196-84de-2856b99cf0ec)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" podUID="532fb4f1-aa6a-4196-84de-2856b99cf0ec" Jul 18 01:11:49.144019 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:11:49.143981 2580 scope.go:117] "RemoveContainer" containerID="0ea3e9f0457149405044a9c3b79a6cf739d0a6d03d612519354a56907ee80502" Jul 18 01:11:49.144531 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:11:49.144201 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z_odh-ai-gateway-infra(532fb4f1-aa6a-4196-84de-2856b99cf0ec)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" podUID="532fb4f1-aa6a-4196-84de-2856b99cf0ec" Jul 18 01:12:00.146026 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:00.145999 2580 scope.go:117] "RemoveContainer" containerID="0ea3e9f0457149405044a9c3b79a6cf739d0a6d03d612519354a56907ee80502" Jul 18 01:12:01.398614 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:01.398583 2580 generic.go:356] "Generic (PLEG): container finished" podID="532fb4f1-aa6a-4196-84de-2856b99cf0ec" containerID="1b1aac4061348ca2b83b04ffffd0854cf776b21d991d2352856480070a56f581" exitCode=1 Jul 18 01:12:01.399063 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:01.398656 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" event={"ID":"532fb4f1-aa6a-4196-84de-2856b99cf0ec","Type":"ContainerDied","Data":"1b1aac4061348ca2b83b04ffffd0854cf776b21d991d2352856480070a56f581"} Jul 18 01:12:01.399063 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:01.398745 2580 scope.go:117] "RemoveContainer" containerID="0ea3e9f0457149405044a9c3b79a6cf739d0a6d03d612519354a56907ee80502" Jul 18 01:12:01.399063 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:01.399032 2580 scope.go:117] "RemoveContainer" containerID="1b1aac4061348ca2b83b04ffffd0854cf776b21d991d2352856480070a56f581" Jul 18 01:12:01.399273 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:12:01.399250 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z_odh-ai-gateway-infra(532fb4f1-aa6a-4196-84de-2856b99cf0ec)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" podUID="532fb4f1-aa6a-4196-84de-2856b99cf0ec" Jul 18 01:12:01.943735 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:01.943695 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" Jul 18 01:12:02.405206 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:02.405175 2580 scope.go:117] "RemoveContainer" containerID="1b1aac4061348ca2b83b04ffffd0854cf776b21d991d2352856480070a56f581" Jul 18 01:12:02.405594 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:12:02.405343 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z_odh-ai-gateway-infra(532fb4f1-aa6a-4196-84de-2856b99cf0ec)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" podUID="532fb4f1-aa6a-4196-84de-2856b99cf0ec" Jul 18 01:12:06.944324 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:06.944283 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" Jul 18 01:12:06.944742 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:06.944727 2580 scope.go:117] "RemoveContainer" containerID="1b1aac4061348ca2b83b04ffffd0854cf776b21d991d2352856480070a56f581" Jul 18 01:12:06.944931 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:12:06.944912 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z_odh-ai-gateway-infra(532fb4f1-aa6a-4196-84de-2856b99cf0ec)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" podUID="532fb4f1-aa6a-4196-84de-2856b99cf0ec" Jul 18 01:12:16.008727 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:16.008682 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z"] Jul 18 01:12:16.147729 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:16.147701 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" Jul 18 01:12:16.176549 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:16.176518 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/532fb4f1-aa6a-4196-84de-2856b99cf0ec-maas-api-tls\") pod \"532fb4f1-aa6a-4196-84de-2856b99cf0ec\" (UID: \"532fb4f1-aa6a-4196-84de-2856b99cf0ec\") " Jul 18 01:12:16.176793 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:16.176589 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/532fb4f1-aa6a-4196-84de-2856b99cf0ec-openshift-service-ca\") pod \"532fb4f1-aa6a-4196-84de-2856b99cf0ec\" (UID: \"532fb4f1-aa6a-4196-84de-2856b99cf0ec\") " Jul 18 01:12:16.176793 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:16.176656 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxnrw\" (UniqueName: \"kubernetes.io/projected/532fb4f1-aa6a-4196-84de-2856b99cf0ec-kube-api-access-lxnrw\") pod \"532fb4f1-aa6a-4196-84de-2856b99cf0ec\" (UID: \"532fb4f1-aa6a-4196-84de-2856b99cf0ec\") " Jul 18 01:12:16.177001 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:16.176971 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/532fb4f1-aa6a-4196-84de-2856b99cf0ec-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "532fb4f1-aa6a-4196-84de-2856b99cf0ec" (UID: "532fb4f1-aa6a-4196-84de-2856b99cf0ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 01:12:16.178914 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:16.178886 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/532fb4f1-aa6a-4196-84de-2856b99cf0ec-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "532fb4f1-aa6a-4196-84de-2856b99cf0ec" (UID: "532fb4f1-aa6a-4196-84de-2856b99cf0ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 01:12:16.178914 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:16.178899 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/532fb4f1-aa6a-4196-84de-2856b99cf0ec-kube-api-access-lxnrw" (OuterVolumeSpecName: "kube-api-access-lxnrw") pod "532fb4f1-aa6a-4196-84de-2856b99cf0ec" (UID: "532fb4f1-aa6a-4196-84de-2856b99cf0ec"). InnerVolumeSpecName "kube-api-access-lxnrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 01:12:16.278056 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:16.278022 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/532fb4f1-aa6a-4196-84de-2856b99cf0ec-maas-api-tls\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:12:16.278056 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:16.278051 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/532fb4f1-aa6a-4196-84de-2856b99cf0ec-openshift-service-ca\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:12:16.278056 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:16.278060 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxnrw\" (UniqueName: \"kubernetes.io/projected/532fb4f1-aa6a-4196-84de-2856b99cf0ec-kube-api-access-lxnrw\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:12:16.474631 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:16.474602 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" Jul 18 01:12:16.474822 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:16.474636 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z" event={"ID":"532fb4f1-aa6a-4196-84de-2856b99cf0ec","Type":"ContainerDied","Data":"cc978588d9f63f7892fa275f4652cab4e52e5019548c9cf5366fae0dfccde9fd"} Jul 18 01:12:16.474822 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:16.474721 2580 scope.go:117] "RemoveContainer" containerID="1b1aac4061348ca2b83b04ffffd0854cf776b21d991d2352856480070a56f581" Jul 18 01:12:16.505934 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:16.505894 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z"] Jul 18 01:12:16.508997 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:16.508968 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1b6716ba-5fc6446b69-88r2z"] Jul 18 01:12:18.150142 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:18.150105 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="532fb4f1-aa6a-4196-84de-2856b99cf0ec" path="/var/lib/kubelet/pods/532fb4f1-aa6a-4196-84de-2856b99cf0ec/volumes" Jul 18 01:12:23.497857 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:23.497815 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg"] Jul 18 01:12:23.498262 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:23.498198 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" containerName="maas-api" Jul 18 01:12:23.498262 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:23.498209 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" containerName="maas-api" Jul 18 01:12:23.498262 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:23.498221 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="532fb4f1-aa6a-4196-84de-2856b99cf0ec" containerName="maas-api" Jul 18 01:12:23.498262 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:23.498227 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="532fb4f1-aa6a-4196-84de-2856b99cf0ec" containerName="maas-api" Jul 18 01:12:23.498262 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:23.498236 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="532fb4f1-aa6a-4196-84de-2856b99cf0ec" containerName="maas-api" Jul 18 01:12:23.498262 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:23.498242 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="532fb4f1-aa6a-4196-84de-2856b99cf0ec" containerName="maas-api" Jul 18 01:12:23.498262 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:23.498248 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="532fb4f1-aa6a-4196-84de-2856b99cf0ec" containerName="maas-api" Jul 18 01:12:23.498262 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:23.498255 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="532fb4f1-aa6a-4196-84de-2856b99cf0ec" containerName="maas-api" Jul 18 01:12:23.498735 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:23.498340 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="532fb4f1-aa6a-4196-84de-2856b99cf0ec" containerName="maas-api" Jul 18 01:12:23.498735 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:23.498352 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="532fb4f1-aa6a-4196-84de-2856b99cf0ec" containerName="maas-api" Jul 18 01:12:23.498735 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:23.498362 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" containerName="maas-api" Jul 18 01:12:23.498735 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:23.498372 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="532fb4f1-aa6a-4196-84de-2856b99cf0ec" containerName="maas-api" Jul 18 01:12:23.498735 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:23.498381 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f2e700e-b1a5-421b-8bac-c15e729b0e9b" containerName="maas-api" Jul 18 01:12:23.503200 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:23.503172 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" Jul 18 01:12:23.505850 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:23.505821 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2djvc\"" Jul 18 01:12:23.506022 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:23.505821 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d5466ad4-serving-cert\"" Jul 18 01:12:23.513640 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:23.513117 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg"] Jul 18 01:12:23.543178 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:23.543141 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8f2576-4655-4e94-bcf1-2a9e4fba7ece-maas-api-tls\") pod \"maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg\" (UID: \"ef8f2576-4655-4e94-bcf1-2a9e4fba7ece\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" Jul 18 01:12:23.543178 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:23.543179 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8f2576-4655-4e94-bcf1-2a9e4fba7ece-openshift-service-ca\") pod \"maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg\" (UID: \"ef8f2576-4655-4e94-bcf1-2a9e4fba7ece\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" Jul 18 01:12:23.543421 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:23.543259 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvdf9\" (UniqueName: \"kubernetes.io/projected/ef8f2576-4655-4e94-bcf1-2a9e4fba7ece-kube-api-access-hvdf9\") pod \"maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg\" (UID: \"ef8f2576-4655-4e94-bcf1-2a9e4fba7ece\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" Jul 18 01:12:23.644553 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:23.644506 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvdf9\" (UniqueName: \"kubernetes.io/projected/ef8f2576-4655-4e94-bcf1-2a9e4fba7ece-kube-api-access-hvdf9\") pod \"maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg\" (UID: \"ef8f2576-4655-4e94-bcf1-2a9e4fba7ece\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" Jul 18 01:12:23.644779 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:23.644609 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8f2576-4655-4e94-bcf1-2a9e4fba7ece-maas-api-tls\") pod \"maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg\" (UID: \"ef8f2576-4655-4e94-bcf1-2a9e4fba7ece\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" Jul 18 01:12:23.644779 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:23.644628 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8f2576-4655-4e94-bcf1-2a9e4fba7ece-openshift-service-ca\") pod \"maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg\" (UID: \"ef8f2576-4655-4e94-bcf1-2a9e4fba7ece\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" Jul 18 01:12:23.644779 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:12:23.644760 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-serving-cert: secret "maas-api-e2e-ait-d5466ad4-serving-cert" not found Jul 18 01:12:23.644916 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:12:23.644829 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef8f2576-4655-4e94-bcf1-2a9e4fba7ece-maas-api-tls podName:ef8f2576-4655-4e94-bcf1-2a9e4fba7ece nodeName:}" failed. No retries permitted until 2026-07-18 01:12:24.144812643 +0000 UTC m=+1904.496662965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ef8f2576-4655-4e94-bcf1-2a9e4fba7ece-maas-api-tls") pod "maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" (UID: "ef8f2576-4655-4e94-bcf1-2a9e4fba7ece") : secret "maas-api-e2e-ait-d5466ad4-serving-cert" not found Jul 18 01:12:23.645271 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:23.645253 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8f2576-4655-4e94-bcf1-2a9e4fba7ece-openshift-service-ca\") pod \"maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg\" (UID: \"ef8f2576-4655-4e94-bcf1-2a9e4fba7ece\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" Jul 18 01:12:23.655131 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:23.655105 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvdf9\" (UniqueName: \"kubernetes.io/projected/ef8f2576-4655-4e94-bcf1-2a9e4fba7ece-kube-api-access-hvdf9\") pod \"maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg\" (UID: \"ef8f2576-4655-4e94-bcf1-2a9e4fba7ece\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" Jul 18 01:12:24.151025 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:24.150993 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8f2576-4655-4e94-bcf1-2a9e4fba7ece-maas-api-tls\") pod \"maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg\" (UID: \"ef8f2576-4655-4e94-bcf1-2a9e4fba7ece\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" Jul 18 01:12:24.153489 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:24.153470 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8f2576-4655-4e94-bcf1-2a9e4fba7ece-maas-api-tls\") pod \"maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg\" (UID: \"ef8f2576-4655-4e94-bcf1-2a9e4fba7ece\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" Jul 18 01:12:24.418772 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:24.418646 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" Jul 18 01:12:24.560958 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:24.560931 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg"] Jul 18 01:12:24.562890 ip-10-0-134-239 kubenswrapper[2580]: W0718 01:12:24.562863 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef8f2576_4655_4e94_bcf1_2a9e4fba7ece.slice/crio-37e295f935118a8fe60c3187878ee2d4314d86dde4d31f05cb4941be2ad3e19a WatchSource:0}: Error finding container 37e295f935118a8fe60c3187878ee2d4314d86dde4d31f05cb4941be2ad3e19a: Status 404 returned error can't find the container with id 37e295f935118a8fe60c3187878ee2d4314d86dde4d31f05cb4941be2ad3e19a Jul 18 01:12:25.532318 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:25.532216 2580 generic.go:356] "Generic (PLEG): container finished" podID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" containerID="ad3a4d4502de2922a59e272fa06caa67bb190c026e026d1aed9b1ea9a6d7dc3f" exitCode=1 Jul 18 01:12:25.532318 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:25.532300 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" event={"ID":"ef8f2576-4655-4e94-bcf1-2a9e4fba7ece","Type":"ContainerDied","Data":"ad3a4d4502de2922a59e272fa06caa67bb190c026e026d1aed9b1ea9a6d7dc3f"} Jul 18 01:12:25.532546 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:25.532350 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" event={"ID":"ef8f2576-4655-4e94-bcf1-2a9e4fba7ece","Type":"ContainerStarted","Data":"37e295f935118a8fe60c3187878ee2d4314d86dde4d31f05cb4941be2ad3e19a"} Jul 18 01:12:25.532607 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:25.532564 2580 scope.go:117] "RemoveContainer" containerID="ad3a4d4502de2922a59e272fa06caa67bb190c026e026d1aed9b1ea9a6d7dc3f" Jul 18 01:12:26.539256 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:26.539173 2580 generic.go:356] "Generic (PLEG): container finished" podID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" containerID="25526c769fa1c71744afff933539a068e97c394d452fd3c329ff5733ddf02263" exitCode=1 Jul 18 01:12:26.539256 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:26.539223 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" event={"ID":"ef8f2576-4655-4e94-bcf1-2a9e4fba7ece","Type":"ContainerDied","Data":"25526c769fa1c71744afff933539a068e97c394d452fd3c329ff5733ddf02263"} Jul 18 01:12:26.539703 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:26.539268 2580 scope.go:117] "RemoveContainer" containerID="ad3a4d4502de2922a59e272fa06caa67bb190c026e026d1aed9b1ea9a6d7dc3f" Jul 18 01:12:26.539703 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:26.539513 2580 scope.go:117] "RemoveContainer" containerID="25526c769fa1c71744afff933539a068e97c394d452fd3c329ff5733ddf02263" Jul 18 01:12:26.539789 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:12:26.539761 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:12:27.547235 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:27.547209 2580 scope.go:117] "RemoveContainer" containerID="25526c769fa1c71744afff933539a068e97c394d452fd3c329ff5733ddf02263" Jul 18 01:12:27.547606 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:12:27.547389 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:12:29.419220 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:29.419187 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" Jul 18 01:12:29.419617 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:29.419556 2580 scope.go:117] "RemoveContainer" containerID="25526c769fa1c71744afff933539a068e97c394d452fd3c329ff5733ddf02263" Jul 18 01:12:29.419783 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:12:29.419761 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:12:34.419140 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:34.419103 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" Jul 18 01:12:34.419536 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:34.419507 2580 scope.go:117] "RemoveContainer" containerID="25526c769fa1c71744afff933539a068e97c394d452fd3c329ff5733ddf02263" Jul 18 01:12:34.419739 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:12:34.419716 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:12:46.143470 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:46.143440 2580 scope.go:117] "RemoveContainer" containerID="25526c769fa1c71744afff933539a068e97c394d452fd3c329ff5733ddf02263" Jul 18 01:12:46.642874 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:46.642830 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" event={"ID":"ef8f2576-4655-4e94-bcf1-2a9e4fba7ece","Type":"ContainerStarted","Data":"3e7239d0149112d98ae07aade0658ad9f2d5b964962299deb877a6121214521d"} Jul 18 01:12:46.643112 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:46.643061 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" Jul 18 01:12:46.659491 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:46.659442 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podStartSLOduration=22.454995773 podStartE2EDuration="23.659426509s" totalImagesPullingTime="1.204430736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:12:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:12:46.65915392 +0000 UTC m=+1927.011004263" watchObservedRunningTime="2026-07-18 01:12:46.659426509 +0000 UTC m=+1927.011276853" Jul 18 01:12:47.650570 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:47.650533 2580 generic.go:356] "Generic (PLEG): container finished" podID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" containerID="3e7239d0149112d98ae07aade0658ad9f2d5b964962299deb877a6121214521d" exitCode=1 Jul 18 01:12:47.651134 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:47.650588 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" event={"ID":"ef8f2576-4655-4e94-bcf1-2a9e4fba7ece","Type":"ContainerDied","Data":"3e7239d0149112d98ae07aade0658ad9f2d5b964962299deb877a6121214521d"} Jul 18 01:12:47.651134 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:47.650651 2580 scope.go:117] "RemoveContainer" containerID="25526c769fa1c71744afff933539a068e97c394d452fd3c329ff5733ddf02263" Jul 18 01:12:47.651134 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:47.650990 2580 scope.go:117] "RemoveContainer" containerID="3e7239d0149112d98ae07aade0658ad9f2d5b964962299deb877a6121214521d" Jul 18 01:12:47.651304 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:12:47.651217 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:12:48.660898 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:48.660872 2580 scope.go:117] "RemoveContainer" containerID="3e7239d0149112d98ae07aade0658ad9f2d5b964962299deb877a6121214521d" Jul 18 01:12:48.661278 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:12:48.661048 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:12:54.419304 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:54.419264 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" Jul 18 01:12:54.419732 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:12:54.419707 2580 scope.go:117] "RemoveContainer" containerID="3e7239d0149112d98ae07aade0658ad9f2d5b964962299deb877a6121214521d" Jul 18 01:12:54.419911 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:12:54.419892 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:13:07.143641 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:13:07.143599 2580 scope.go:117] "RemoveContainer" containerID="3e7239d0149112d98ae07aade0658ad9f2d5b964962299deb877a6121214521d" Jul 18 01:13:07.144057 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:13:07.143962 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 01:13:07.757368 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:13:07.757336 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" event={"ID":"ef8f2576-4655-4e94-bcf1-2a9e4fba7ece","Type":"ContainerStarted","Data":"b8f15f149882931269014dc7d161c44682c1918ccf61720afc097ca6dcc7e7a3"} Jul 18 01:13:07.757567 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:13:07.757549 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" Jul 18 01:13:08.764493 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:13:08.764460 2580 generic.go:356] "Generic (PLEG): container finished" podID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" containerID="b8f15f149882931269014dc7d161c44682c1918ccf61720afc097ca6dcc7e7a3" exitCode=1 Jul 18 01:13:08.764940 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:13:08.764559 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" event={"ID":"ef8f2576-4655-4e94-bcf1-2a9e4fba7ece","Type":"ContainerDied","Data":"b8f15f149882931269014dc7d161c44682c1918ccf61720afc097ca6dcc7e7a3"} Jul 18 01:13:08.764940 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:13:08.764620 2580 scope.go:117] "RemoveContainer" containerID="3e7239d0149112d98ae07aade0658ad9f2d5b964962299deb877a6121214521d" Jul 18 01:13:08.764940 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:13:08.764859 2580 scope.go:117] "RemoveContainer" containerID="b8f15f149882931269014dc7d161c44682c1918ccf61720afc097ca6dcc7e7a3" Jul 18 01:13:08.765067 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:13:08.765050 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:13:09.771363 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:13:09.771334 2580 scope.go:117] "RemoveContainer" containerID="b8f15f149882931269014dc7d161c44682c1918ccf61720afc097ca6dcc7e7a3" Jul 18 01:13:09.771755 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:13:09.771522 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:13:14.419235 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:13:14.419194 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" Jul 18 01:13:14.419633 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:13:14.419565 2580 scope.go:117] "RemoveContainer" containerID="b8f15f149882931269014dc7d161c44682c1918ccf61720afc097ca6dcc7e7a3" Jul 18 01:13:14.419791 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:13:14.419771 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:13:29.143907 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:13:29.143867 2580 scope.go:117] "RemoveContainer" containerID="b8f15f149882931269014dc7d161c44682c1918ccf61720afc097ca6dcc7e7a3" Jul 18 01:13:29.144508 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:13:29.144121 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:13:43.144165 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:13:43.144087 2580 scope.go:117] "RemoveContainer" containerID="b8f15f149882931269014dc7d161c44682c1918ccf61720afc097ca6dcc7e7a3" Jul 18 01:13:43.144528 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:13:43.144281 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:13:58.144395 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:13:58.144362 2580 scope.go:117] "RemoveContainer" containerID="b8f15f149882931269014dc7d161c44682c1918ccf61720afc097ca6dcc7e7a3" Jul 18 01:13:59.018330 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:13:59.018293 2580 generic.go:356] "Generic (PLEG): container finished" podID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" containerID="3fd6adf3d56f713e91f4d8edb6beb0100e85294613c96979e7fad60e3e368c6f" exitCode=1 Jul 18 01:13:59.018498 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:13:59.018367 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" event={"ID":"ef8f2576-4655-4e94-bcf1-2a9e4fba7ece","Type":"ContainerDied","Data":"3fd6adf3d56f713e91f4d8edb6beb0100e85294613c96979e7fad60e3e368c6f"} Jul 18 01:13:59.018498 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:13:59.018430 2580 scope.go:117] "RemoveContainer" containerID="b8f15f149882931269014dc7d161c44682c1918ccf61720afc097ca6dcc7e7a3" Jul 18 01:13:59.018827 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:13:59.018809 2580 scope.go:117] "RemoveContainer" containerID="3fd6adf3d56f713e91f4d8edb6beb0100e85294613c96979e7fad60e3e368c6f" Jul 18 01:13:59.019031 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:13:59.019012 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:14:02.758502 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:14:02.758463 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" Jul 18 01:14:02.759032 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:14:02.758823 2580 scope.go:117] "RemoveContainer" containerID="3fd6adf3d56f713e91f4d8edb6beb0100e85294613c96979e7fad60e3e368c6f" Jul 18 01:14:02.759032 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:14:02.759016 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:14:04.419388 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:14:04.419335 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" Jul 18 01:14:04.419918 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:14:04.419877 2580 scope.go:117] "RemoveContainer" containerID="3fd6adf3d56f713e91f4d8edb6beb0100e85294613c96979e7fad60e3e368c6f" Jul 18 01:14:04.420124 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:14:04.420100 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:14:18.144190 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:14:18.144160 2580 scope.go:117] "RemoveContainer" containerID="3fd6adf3d56f713e91f4d8edb6beb0100e85294613c96979e7fad60e3e368c6f" Jul 18 01:14:18.144590 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:14:18.144329 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:14:30.146386 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:14:30.146355 2580 scope.go:117] "RemoveContainer" containerID="3fd6adf3d56f713e91f4d8edb6beb0100e85294613c96979e7fad60e3e368c6f" Jul 18 01:14:30.146826 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:14:30.146572 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:14:45.143918 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:14:45.143888 2580 scope.go:117] "RemoveContainer" containerID="3fd6adf3d56f713e91f4d8edb6beb0100e85294613c96979e7fad60e3e368c6f" Jul 18 01:14:45.144382 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:14:45.144058 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:15:00.149349 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:15:00.149317 2580 scope.go:117] "RemoveContainer" containerID="3fd6adf3d56f713e91f4d8edb6beb0100e85294613c96979e7fad60e3e368c6f" Jul 18 01:15:00.149747 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:15:00.149492 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:15:14.143814 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:15:14.143785 2580 scope.go:117] "RemoveContainer" containerID="3fd6adf3d56f713e91f4d8edb6beb0100e85294613c96979e7fad60e3e368c6f" Jul 18 01:15:14.144303 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:15:14.143975 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:15:25.143817 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:15:25.143779 2580 scope.go:117] "RemoveContainer" containerID="3fd6adf3d56f713e91f4d8edb6beb0100e85294613c96979e7fad60e3e368c6f" Jul 18 01:15:26.458500 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:15:26.458468 2580 generic.go:356] "Generic (PLEG): container finished" podID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" containerID="70fb8427e866424f8e99a75ed6d9b701c1e5d55a020f7bca38b1ac586bfe2b8b" exitCode=1 Jul 18 01:15:26.458928 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:15:26.458542 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" event={"ID":"ef8f2576-4655-4e94-bcf1-2a9e4fba7ece","Type":"ContainerDied","Data":"70fb8427e866424f8e99a75ed6d9b701c1e5d55a020f7bca38b1ac586bfe2b8b"} Jul 18 01:15:26.458928 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:15:26.458604 2580 scope.go:117] "RemoveContainer" containerID="3fd6adf3d56f713e91f4d8edb6beb0100e85294613c96979e7fad60e3e368c6f" Jul 18 01:15:26.458928 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:15:26.458913 2580 scope.go:117] "RemoveContainer" containerID="70fb8427e866424f8e99a75ed6d9b701c1e5d55a020f7bca38b1ac586bfe2b8b" Jul 18 01:15:26.459150 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:15:26.459128 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:15:27.757778 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:15:27.757738 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" Jul 18 01:15:27.758170 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:15:27.758103 2580 scope.go:117] "RemoveContainer" containerID="70fb8427e866424f8e99a75ed6d9b701c1e5d55a020f7bca38b1ac586bfe2b8b" Jul 18 01:15:27.758323 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:15:27.758304 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:15:34.418924 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:15:34.418888 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" Jul 18 01:15:34.419324 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:15:34.419253 2580 scope.go:117] "RemoveContainer" containerID="70fb8427e866424f8e99a75ed6d9b701c1e5d55a020f7bca38b1ac586bfe2b8b" Jul 18 01:15:34.419458 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:15:34.419438 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:15:48.144371 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:15:48.144335 2580 scope.go:117] "RemoveContainer" containerID="70fb8427e866424f8e99a75ed6d9b701c1e5d55a020f7bca38b1ac586bfe2b8b" Jul 18 01:15:48.144780 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:15:48.144493 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:15:59.144188 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:15:59.144147 2580 scope.go:117] "RemoveContainer" containerID="70fb8427e866424f8e99a75ed6d9b701c1e5d55a020f7bca38b1ac586bfe2b8b" Jul 18 01:15:59.144615 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:15:59.144317 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:16:10.148018 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:16:10.147974 2580 scope.go:117] "RemoveContainer" containerID="70fb8427e866424f8e99a75ed6d9b701c1e5d55a020f7bca38b1ac586bfe2b8b" Jul 18 01:16:10.148433 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:16:10.148189 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:16:23.143831 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:16:23.143792 2580 scope.go:117] "RemoveContainer" containerID="70fb8427e866424f8e99a75ed6d9b701c1e5d55a020f7bca38b1ac586bfe2b8b" Jul 18 01:16:23.144366 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:16:23.144037 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:16:37.143809 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:16:37.143722 2580 scope.go:117] "RemoveContainer" containerID="70fb8427e866424f8e99a75ed6d9b701c1e5d55a020f7bca38b1ac586bfe2b8b" Jul 18 01:16:37.144298 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:16:37.143932 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:16:49.144329 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:16:49.144288 2580 scope.go:117] "RemoveContainer" containerID="70fb8427e866424f8e99a75ed6d9b701c1e5d55a020f7bca38b1ac586bfe2b8b" Jul 18 01:16:49.144883 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:16:49.144480 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:17:00.146907 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:17:00.146866 2580 scope.go:117] "RemoveContainer" containerID="70fb8427e866424f8e99a75ed6d9b701c1e5d55a020f7bca38b1ac586bfe2b8b" Jul 18 01:17:00.147371 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:17:00.147068 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:17:11.143933 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:17:11.143897 2580 scope.go:117] "RemoveContainer" containerID="70fb8427e866424f8e99a75ed6d9b701c1e5d55a020f7bca38b1ac586bfe2b8b" Jul 18 01:17:11.144433 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:17:11.144072 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:17:26.143657 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:17:26.143620 2580 scope.go:117] "RemoveContainer" containerID="70fb8427e866424f8e99a75ed6d9b701c1e5d55a020f7bca38b1ac586bfe2b8b" Jul 18 01:17:26.144145 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:17:26.143896 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:17:40.147037 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:17:40.146994 2580 scope.go:117] "RemoveContainer" containerID="70fb8427e866424f8e99a75ed6d9b701c1e5d55a020f7bca38b1ac586bfe2b8b" Jul 18 01:17:40.147551 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:17:40.147219 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:17:55.143925 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:17:55.143885 2580 scope.go:117] "RemoveContainer" containerID="70fb8427e866424f8e99a75ed6d9b701c1e5d55a020f7bca38b1ac586bfe2b8b" Jul 18 01:17:55.146219 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:17:55.144098 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:18:06.144497 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:06.144397 2580 scope.go:117] "RemoveContainer" containerID="70fb8427e866424f8e99a75ed6d9b701c1e5d55a020f7bca38b1ac586bfe2b8b" Jul 18 01:18:07.264405 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:07.264373 2580 generic.go:356] "Generic (PLEG): container finished" podID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" containerID="b60c09e20d5d8101d89c6878c0a2237512fb5a0107c2158b47a26df90e91dc78" exitCode=1 Jul 18 01:18:07.264904 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:07.264443 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" event={"ID":"ef8f2576-4655-4e94-bcf1-2a9e4fba7ece","Type":"ContainerDied","Data":"b60c09e20d5d8101d89c6878c0a2237512fb5a0107c2158b47a26df90e91dc78"} Jul 18 01:18:07.264904 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:07.264505 2580 scope.go:117] "RemoveContainer" containerID="70fb8427e866424f8e99a75ed6d9b701c1e5d55a020f7bca38b1ac586bfe2b8b" Jul 18 01:18:07.264904 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:07.264880 2580 scope.go:117] "RemoveContainer" containerID="b60c09e20d5d8101d89c6878c0a2237512fb5a0107c2158b47a26df90e91dc78" Jul 18 01:18:07.265117 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:18:07.265097 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:18:07.758015 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:07.757988 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" Jul 18 01:18:08.271600 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:08.271575 2580 scope.go:117] "RemoveContainer" containerID="b60c09e20d5d8101d89c6878c0a2237512fb5a0107c2158b47a26df90e91dc78" Jul 18 01:18:08.272029 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:18:08.271773 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:18:14.418998 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:14.418953 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" Jul 18 01:18:14.419476 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:14.419347 2580 scope.go:117] "RemoveContainer" containerID="b60c09e20d5d8101d89c6878c0a2237512fb5a0107c2158b47a26df90e91dc78" Jul 18 01:18:14.419568 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:18:14.419546 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:18:27.144320 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:27.144291 2580 scope.go:117] "RemoveContainer" containerID="b60c09e20d5d8101d89c6878c0a2237512fb5a0107c2158b47a26df90e91dc78" Jul 18 01:18:27.144827 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:18:27.144458 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg_odh-ai-gateway-infra(ef8f2576-4655-4e94-bcf1-2a9e4fba7ece)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" Jul 18 01:18:33.739287 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:33.739251 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg"] Jul 18 01:18:33.871481 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:33.871450 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" Jul 18 01:18:33.906542 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:33.906508 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8f2576-4655-4e94-bcf1-2a9e4fba7ece-openshift-service-ca\") pod \"ef8f2576-4655-4e94-bcf1-2a9e4fba7ece\" (UID: \"ef8f2576-4655-4e94-bcf1-2a9e4fba7ece\") " Jul 18 01:18:33.906752 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:33.906557 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8f2576-4655-4e94-bcf1-2a9e4fba7ece-maas-api-tls\") pod \"ef8f2576-4655-4e94-bcf1-2a9e4fba7ece\" (UID: \"ef8f2576-4655-4e94-bcf1-2a9e4fba7ece\") " Jul 18 01:18:33.906752 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:33.906582 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvdf9\" (UniqueName: \"kubernetes.io/projected/ef8f2576-4655-4e94-bcf1-2a9e4fba7ece-kube-api-access-hvdf9\") pod \"ef8f2576-4655-4e94-bcf1-2a9e4fba7ece\" (UID: \"ef8f2576-4655-4e94-bcf1-2a9e4fba7ece\") " Jul 18 01:18:33.907034 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:33.907008 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef8f2576-4655-4e94-bcf1-2a9e4fba7ece-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" (UID: "ef8f2576-4655-4e94-bcf1-2a9e4fba7ece"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 01:18:33.909025 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:33.908993 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef8f2576-4655-4e94-bcf1-2a9e4fba7ece-kube-api-access-hvdf9" (OuterVolumeSpecName: "kube-api-access-hvdf9") pod "ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" (UID: "ef8f2576-4655-4e94-bcf1-2a9e4fba7ece"). InnerVolumeSpecName "kube-api-access-hvdf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 01:18:33.909139 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:33.909047 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef8f2576-4655-4e94-bcf1-2a9e4fba7ece-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" (UID: "ef8f2576-4655-4e94-bcf1-2a9e4fba7ece"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 01:18:34.007361 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:34.007270 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8f2576-4655-4e94-bcf1-2a9e4fba7ece-openshift-service-ca\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:18:34.007361 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:34.007302 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef8f2576-4655-4e94-bcf1-2a9e4fba7ece-maas-api-tls\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:18:34.007361 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:34.007310 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvdf9\" (UniqueName: \"kubernetes.io/projected/ef8f2576-4655-4e94-bcf1-2a9e4fba7ece-kube-api-access-hvdf9\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:18:34.405650 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:34.405561 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" Jul 18 01:18:34.405650 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:34.405571 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg" event={"ID":"ef8f2576-4655-4e94-bcf1-2a9e4fba7ece","Type":"ContainerDied","Data":"37e295f935118a8fe60c3187878ee2d4314d86dde4d31f05cb4941be2ad3e19a"} Jul 18 01:18:34.405650 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:34.405627 2580 scope.go:117] "RemoveContainer" containerID="b60c09e20d5d8101d89c6878c0a2237512fb5a0107c2158b47a26df90e91dc78" Jul 18 01:18:34.422399 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:34.422366 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg"] Jul 18 01:18:34.426202 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:34.426175 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d5466ad4-b9f8b8c88-mzpdg"] Jul 18 01:18:36.149753 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:36.149717 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" path="/var/lib/kubelet/pods/ef8f2576-4655-4e94-bcf1-2a9e4fba7ece/volumes" Jul 18 01:18:41.530943 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.530903 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w"] Jul 18 01:18:41.531466 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.531445 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="532fb4f1-aa6a-4196-84de-2856b99cf0ec" containerName="maas-api" Jul 18 01:18:41.531521 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.531472 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="532fb4f1-aa6a-4196-84de-2856b99cf0ec" containerName="maas-api" Jul 18 01:18:41.531521 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.531501 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" containerName="maas-api" Jul 18 01:18:41.531521 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.531510 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" containerName="maas-api" Jul 18 01:18:41.531625 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.531526 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" containerName="maas-api" Jul 18 01:18:41.531625 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.531535 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" containerName="maas-api" Jul 18 01:18:41.531625 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.531546 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" containerName="maas-api" Jul 18 01:18:41.531625 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.531555 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" containerName="maas-api" Jul 18 01:18:41.531625 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.531568 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" containerName="maas-api" Jul 18 01:18:41.531625 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.531577 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" containerName="maas-api" Jul 18 01:18:41.531625 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.531589 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" containerName="maas-api" Jul 18 01:18:41.531625 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.531597 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" containerName="maas-api" Jul 18 01:18:41.531900 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.531718 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" containerName="maas-api" Jul 18 01:18:41.531900 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.531730 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" containerName="maas-api" Jul 18 01:18:41.531900 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.531740 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" containerName="maas-api" Jul 18 01:18:41.531900 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.531751 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" containerName="maas-api" Jul 18 01:18:41.531900 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.531761 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" containerName="maas-api" Jul 18 01:18:41.531900 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.531769 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" containerName="maas-api" Jul 18 01:18:41.531900 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.531778 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="532fb4f1-aa6a-4196-84de-2856b99cf0ec" containerName="maas-api" Jul 18 01:18:41.536991 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.536962 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" Jul 18 01:18:41.539496 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.539465 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-8942d9e2-serving-cert\"" Jul 18 01:18:41.540859 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.540837 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2djvc\"" Jul 18 01:18:41.543182 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.543117 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w"] Jul 18 01:18:41.576518 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.576478 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/695f0f72-fba2-4a27-8c91-9db7b64b351b-maas-api-tls\") pod \"maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w\" (UID: \"695f0f72-fba2-4a27-8c91-9db7b64b351b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" Jul 18 01:18:41.576778 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.576629 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/695f0f72-fba2-4a27-8c91-9db7b64b351b-openshift-service-ca\") pod \"maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w\" (UID: \"695f0f72-fba2-4a27-8c91-9db7b64b351b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" Jul 18 01:18:41.576778 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.576712 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc6jf\" (UniqueName: \"kubernetes.io/projected/695f0f72-fba2-4a27-8c91-9db7b64b351b-kube-api-access-gc6jf\") pod \"maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w\" (UID: \"695f0f72-fba2-4a27-8c91-9db7b64b351b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" Jul 18 01:18:41.677394 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.677353 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/695f0f72-fba2-4a27-8c91-9db7b64b351b-openshift-service-ca\") pod \"maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w\" (UID: \"695f0f72-fba2-4a27-8c91-9db7b64b351b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" Jul 18 01:18:41.677394 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.677398 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc6jf\" (UniqueName: \"kubernetes.io/projected/695f0f72-fba2-4a27-8c91-9db7b64b351b-kube-api-access-gc6jf\") pod \"maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w\" (UID: \"695f0f72-fba2-4a27-8c91-9db7b64b351b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" Jul 18 01:18:41.677619 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.677437 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/695f0f72-fba2-4a27-8c91-9db7b64b351b-maas-api-tls\") pod \"maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w\" (UID: \"695f0f72-fba2-4a27-8c91-9db7b64b351b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" Jul 18 01:18:41.677619 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:18:41.677565 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-serving-cert: secret "maas-api-e2e-derive-8942d9e2-serving-cert" not found Jul 18 01:18:41.677734 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:18:41.677638 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/695f0f72-fba2-4a27-8c91-9db7b64b351b-maas-api-tls podName:695f0f72-fba2-4a27-8c91-9db7b64b351b nodeName:}" failed. No retries permitted until 2026-07-18 01:18:42.177619404 +0000 UTC m=+2282.529469726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/695f0f72-fba2-4a27-8c91-9db7b64b351b-maas-api-tls") pod "maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" (UID: "695f0f72-fba2-4a27-8c91-9db7b64b351b") : secret "maas-api-e2e-derive-8942d9e2-serving-cert" not found Jul 18 01:18:41.678151 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.678131 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/695f0f72-fba2-4a27-8c91-9db7b64b351b-openshift-service-ca\") pod \"maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w\" (UID: \"695f0f72-fba2-4a27-8c91-9db7b64b351b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" Jul 18 01:18:41.687051 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:41.687022 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc6jf\" (UniqueName: \"kubernetes.io/projected/695f0f72-fba2-4a27-8c91-9db7b64b351b-kube-api-access-gc6jf\") pod \"maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w\" (UID: \"695f0f72-fba2-4a27-8c91-9db7b64b351b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" Jul 18 01:18:42.182109 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:42.182018 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/695f0f72-fba2-4a27-8c91-9db7b64b351b-maas-api-tls\") pod \"maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w\" (UID: \"695f0f72-fba2-4a27-8c91-9db7b64b351b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" Jul 18 01:18:42.184643 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:42.184610 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/695f0f72-fba2-4a27-8c91-9db7b64b351b-maas-api-tls\") pod \"maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w\" (UID: \"695f0f72-fba2-4a27-8c91-9db7b64b351b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" Jul 18 01:18:42.450927 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:42.450834 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" Jul 18 01:18:42.601028 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:42.601001 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w"] Jul 18 01:18:42.603781 ip-10-0-134-239 kubenswrapper[2580]: W0718 01:18:42.603743 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695f0f72_fba2_4a27_8c91_9db7b64b351b.slice/crio-c7fde728e9313a93c7459346e14badda971dfba0067b86a19284c976271ea0de WatchSource:0}: Error finding container c7fde728e9313a93c7459346e14badda971dfba0067b86a19284c976271ea0de: Status 404 returned error can't find the container with id c7fde728e9313a93c7459346e14badda971dfba0067b86a19284c976271ea0de Jul 18 01:18:42.605299 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:42.605278 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 01:18:43.453786 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:43.453751 2580 generic.go:356] "Generic (PLEG): container finished" podID="695f0f72-fba2-4a27-8c91-9db7b64b351b" containerID="e39b0a0b30b58aae992d6b627e33b94b795dba6e5cd330aa9a652d9e8732c8c4" exitCode=1 Jul 18 01:18:43.453972 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:43.453844 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" event={"ID":"695f0f72-fba2-4a27-8c91-9db7b64b351b","Type":"ContainerDied","Data":"e39b0a0b30b58aae992d6b627e33b94b795dba6e5cd330aa9a652d9e8732c8c4"} Jul 18 01:18:43.453972 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:43.453910 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" event={"ID":"695f0f72-fba2-4a27-8c91-9db7b64b351b","Type":"ContainerStarted","Data":"c7fde728e9313a93c7459346e14badda971dfba0067b86a19284c976271ea0de"} Jul 18 01:18:43.454056 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:43.454032 2580 scope.go:117] "RemoveContainer" containerID="e39b0a0b30b58aae992d6b627e33b94b795dba6e5cd330aa9a652d9e8732c8c4" Jul 18 01:18:44.460696 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:44.460575 2580 generic.go:356] "Generic (PLEG): container finished" podID="695f0f72-fba2-4a27-8c91-9db7b64b351b" containerID="d086ee182d123bab1ef2f6c6ba0f004d10e8bf8063c01c3fee70867c58b8a6ff" exitCode=1 Jul 18 01:18:44.460696 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:44.460686 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" event={"ID":"695f0f72-fba2-4a27-8c91-9db7b64b351b","Type":"ContainerDied","Data":"d086ee182d123bab1ef2f6c6ba0f004d10e8bf8063c01c3fee70867c58b8a6ff"} Jul 18 01:18:44.461198 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:44.460746 2580 scope.go:117] "RemoveContainer" containerID="e39b0a0b30b58aae992d6b627e33b94b795dba6e5cd330aa9a652d9e8732c8c4" Jul 18 01:18:44.461198 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:44.461018 2580 scope.go:117] "RemoveContainer" containerID="d086ee182d123bab1ef2f6c6ba0f004d10e8bf8063c01c3fee70867c58b8a6ff" Jul 18 01:18:44.461297 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:18:44.461215 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w_odh-ai-gateway-infra(695f0f72-fba2-4a27-8c91-9db7b64b351b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" Jul 18 01:18:45.467788 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:45.467755 2580 scope.go:117] "RemoveContainer" containerID="d086ee182d123bab1ef2f6c6ba0f004d10e8bf8063c01c3fee70867c58b8a6ff" Jul 18 01:18:45.468154 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:18:45.467942 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w_odh-ai-gateway-infra(695f0f72-fba2-4a27-8c91-9db7b64b351b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" Jul 18 01:18:47.451591 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:47.451546 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" Jul 18 01:18:47.452019 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:47.451982 2580 scope.go:117] "RemoveContainer" containerID="d086ee182d123bab1ef2f6c6ba0f004d10e8bf8063c01c3fee70867c58b8a6ff" Jul 18 01:18:47.452194 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:18:47.452172 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w_odh-ai-gateway-infra(695f0f72-fba2-4a27-8c91-9db7b64b351b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" Jul 18 01:18:52.451203 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:52.451158 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" Jul 18 01:18:52.451574 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:18:52.451541 2580 scope.go:117] "RemoveContainer" containerID="d086ee182d123bab1ef2f6c6ba0f004d10e8bf8063c01c3fee70867c58b8a6ff" Jul 18 01:18:52.451784 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:18:52.451763 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w_odh-ai-gateway-infra(695f0f72-fba2-4a27-8c91-9db7b64b351b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" Jul 18 01:19:06.144339 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:19:06.144305 2580 scope.go:117] "RemoveContainer" containerID="d086ee182d123bab1ef2f6c6ba0f004d10e8bf8063c01c3fee70867c58b8a6ff" Jul 18 01:19:07.575075 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:19:07.575044 2580 generic.go:356] "Generic (PLEG): container finished" podID="695f0f72-fba2-4a27-8c91-9db7b64b351b" containerID="c44a0d8fd22575b94556770d4932d0b4c1c004276be7983d26506181acbfcba7" exitCode=1 Jul 18 01:19:07.575461 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:19:07.575124 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" event={"ID":"695f0f72-fba2-4a27-8c91-9db7b64b351b","Type":"ContainerDied","Data":"c44a0d8fd22575b94556770d4932d0b4c1c004276be7983d26506181acbfcba7"} Jul 18 01:19:07.575461 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:19:07.575183 2580 scope.go:117] "RemoveContainer" containerID="d086ee182d123bab1ef2f6c6ba0f004d10e8bf8063c01c3fee70867c58b8a6ff" Jul 18 01:19:07.575578 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:19:07.575511 2580 scope.go:117] "RemoveContainer" containerID="c44a0d8fd22575b94556770d4932d0b4c1c004276be7983d26506181acbfcba7" Jul 18 01:19:07.575732 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:19:07.575712 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w_odh-ai-gateway-infra(695f0f72-fba2-4a27-8c91-9db7b64b351b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" Jul 18 01:19:12.451770 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:19:12.451729 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" Jul 18 01:19:12.452177 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:19:12.451801 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" Jul 18 01:19:12.452177 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:19:12.452160 2580 scope.go:117] "RemoveContainer" containerID="c44a0d8fd22575b94556770d4932d0b4c1c004276be7983d26506181acbfcba7" Jul 18 01:19:12.452351 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:19:12.452331 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w_odh-ai-gateway-infra(695f0f72-fba2-4a27-8c91-9db7b64b351b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" Jul 18 01:19:23.143896 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:19:23.143863 2580 scope.go:117] "RemoveContainer" containerID="c44a0d8fd22575b94556770d4932d0b4c1c004276be7983d26506181acbfcba7" Jul 18 01:19:23.144385 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:19:23.144039 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w_odh-ai-gateway-infra(695f0f72-fba2-4a27-8c91-9db7b64b351b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" Jul 18 01:19:34.143974 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:19:34.143890 2580 scope.go:117] "RemoveContainer" containerID="c44a0d8fd22575b94556770d4932d0b4c1c004276be7983d26506181acbfcba7" Jul 18 01:19:34.714613 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:19:34.714575 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" event={"ID":"695f0f72-fba2-4a27-8c91-9db7b64b351b","Type":"ContainerStarted","Data":"d5506ae3366046485273f7f56aefcc08521d3e7bd7306bca16ae26bad366cecb"} Jul 18 01:19:34.714841 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:19:34.714822 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" Jul 18 01:19:34.732430 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:19:34.732381 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" podStartSLOduration=52.18532649 podStartE2EDuration="53.732366144s" totalImagesPullingTime="1.547039654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:18:41 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:19:34.729912149 +0000 UTC m=+2335.081762494" watchObservedRunningTime="2026-07-18 01:19:34.732366144 +0000 UTC m=+2335.084216488" Jul 18 01:19:35.721787 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:19:35.721746 2580 generic.go:356] "Generic (PLEG): container finished" podID="695f0f72-fba2-4a27-8c91-9db7b64b351b" containerID="d5506ae3366046485273f7f56aefcc08521d3e7bd7306bca16ae26bad366cecb" exitCode=1 Jul 18 01:19:35.722206 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:19:35.721843 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" event={"ID":"695f0f72-fba2-4a27-8c91-9db7b64b351b","Type":"ContainerDied","Data":"d5506ae3366046485273f7f56aefcc08521d3e7bd7306bca16ae26bad366cecb"} Jul 18 01:19:35.722206 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:19:35.721902 2580 scope.go:117] "RemoveContainer" containerID="c44a0d8fd22575b94556770d4932d0b4c1c004276be7983d26506181acbfcba7" Jul 18 01:19:35.722206 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:19:35.722174 2580 scope.go:117] "RemoveContainer" containerID="d5506ae3366046485273f7f56aefcc08521d3e7bd7306bca16ae26bad366cecb" Jul 18 01:19:35.722397 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:19:35.722379 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w_odh-ai-gateway-infra(695f0f72-fba2-4a27-8c91-9db7b64b351b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" Jul 18 01:19:36.728439 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:19:36.728410 2580 scope.go:117] "RemoveContainer" containerID="d5506ae3366046485273f7f56aefcc08521d3e7bd7306bca16ae26bad366cecb" Jul 18 01:19:36.728843 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:19:36.728588 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w_odh-ai-gateway-infra(695f0f72-fba2-4a27-8c91-9db7b64b351b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" Jul 18 01:19:42.451248 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:19:42.451210 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" Jul 18 01:19:42.451764 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:19:42.451634 2580 scope.go:117] "RemoveContainer" containerID="d5506ae3366046485273f7f56aefcc08521d3e7bd7306bca16ae26bad366cecb" Jul 18 01:19:42.451874 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:19:42.451853 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w_odh-ai-gateway-infra(695f0f72-fba2-4a27-8c91-9db7b64b351b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" Jul 18 01:19:55.143972 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:19:55.143940 2580 scope.go:117] "RemoveContainer" containerID="d5506ae3366046485273f7f56aefcc08521d3e7bd7306bca16ae26bad366cecb" Jul 18 01:19:55.144360 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:19:55.144121 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w_odh-ai-gateway-infra(695f0f72-fba2-4a27-8c91-9db7b64b351b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" Jul 18 01:20:07.144155 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:20:07.144122 2580 scope.go:117] "RemoveContainer" containerID="d5506ae3366046485273f7f56aefcc08521d3e7bd7306bca16ae26bad366cecb" Jul 18 01:20:07.144569 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:20:07.144294 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w_odh-ai-gateway-infra(695f0f72-fba2-4a27-8c91-9db7b64b351b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" Jul 18 01:20:22.143968 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:20:22.143936 2580 scope.go:117] "RemoveContainer" containerID="d5506ae3366046485273f7f56aefcc08521d3e7bd7306bca16ae26bad366cecb" Jul 18 01:20:22.959067 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:20:22.958977 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" event={"ID":"695f0f72-fba2-4a27-8c91-9db7b64b351b","Type":"ContainerStarted","Data":"ad7acb0c9b119f6cbd0fd7c8448d95ff0c21bac8e742b9fd623f708f2c0e4daf"} Jul 18 01:20:22.959225 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:20:22.959190 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" Jul 18 01:20:23.966847 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:20:23.966808 2580 generic.go:356] "Generic (PLEG): container finished" podID="695f0f72-fba2-4a27-8c91-9db7b64b351b" containerID="ad7acb0c9b119f6cbd0fd7c8448d95ff0c21bac8e742b9fd623f708f2c0e4daf" exitCode=1 Jul 18 01:20:23.967304 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:20:23.966893 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" event={"ID":"695f0f72-fba2-4a27-8c91-9db7b64b351b","Type":"ContainerDied","Data":"ad7acb0c9b119f6cbd0fd7c8448d95ff0c21bac8e742b9fd623f708f2c0e4daf"} Jul 18 01:20:23.967304 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:20:23.966946 2580 scope.go:117] "RemoveContainer" containerID="d5506ae3366046485273f7f56aefcc08521d3e7bd7306bca16ae26bad366cecb" Jul 18 01:20:23.967304 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:20:23.967260 2580 scope.go:117] "RemoveContainer" containerID="ad7acb0c9b119f6cbd0fd7c8448d95ff0c21bac8e742b9fd623f708f2c0e4daf" Jul 18 01:20:23.967464 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:20:23.967442 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w_odh-ai-gateway-infra(695f0f72-fba2-4a27-8c91-9db7b64b351b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" Jul 18 01:20:24.974324 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:20:24.974299 2580 scope.go:117] "RemoveContainer" containerID="ad7acb0c9b119f6cbd0fd7c8448d95ff0c21bac8e742b9fd623f708f2c0e4daf" Jul 18 01:20:24.974724 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:20:24.974483 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w_odh-ai-gateway-infra(695f0f72-fba2-4a27-8c91-9db7b64b351b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" Jul 18 01:20:32.451977 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:20:32.451922 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" Jul 18 01:20:32.452417 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:20:32.452398 2580 scope.go:117] "RemoveContainer" containerID="ad7acb0c9b119f6cbd0fd7c8448d95ff0c21bac8e742b9fd623f708f2c0e4daf" Jul 18 01:20:32.452604 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:20:32.452584 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w_odh-ai-gateway-infra(695f0f72-fba2-4a27-8c91-9db7b64b351b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" Jul 18 01:20:48.143891 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:20:48.143861 2580 scope.go:117] "RemoveContainer" containerID="ad7acb0c9b119f6cbd0fd7c8448d95ff0c21bac8e742b9fd623f708f2c0e4daf" Jul 18 01:20:48.144315 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:20:48.144030 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w_odh-ai-gateway-infra(695f0f72-fba2-4a27-8c91-9db7b64b351b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" Jul 18 01:21:03.143932 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:21:03.143848 2580 scope.go:117] "RemoveContainer" containerID="ad7acb0c9b119f6cbd0fd7c8448d95ff0c21bac8e742b9fd623f708f2c0e4daf" Jul 18 01:21:03.144345 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:21:03.144049 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w_odh-ai-gateway-infra(695f0f72-fba2-4a27-8c91-9db7b64b351b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" Jul 18 01:21:16.143832 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:21:16.143792 2580 scope.go:117] "RemoveContainer" containerID="ad7acb0c9b119f6cbd0fd7c8448d95ff0c21bac8e742b9fd623f708f2c0e4daf" Jul 18 01:21:16.144345 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:21:16.143993 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w_odh-ai-gateway-infra(695f0f72-fba2-4a27-8c91-9db7b64b351b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" Jul 18 01:21:30.146340 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:21:30.146301 2580 scope.go:117] "RemoveContainer" containerID="ad7acb0c9b119f6cbd0fd7c8448d95ff0c21bac8e742b9fd623f708f2c0e4daf" Jul 18 01:21:30.146970 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:21:30.146517 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w_odh-ai-gateway-infra(695f0f72-fba2-4a27-8c91-9db7b64b351b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" Jul 18 01:21:44.143308 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:21:44.143277 2580 scope.go:117] "RemoveContainer" containerID="ad7acb0c9b119f6cbd0fd7c8448d95ff0c21bac8e742b9fd623f708f2c0e4daf" Jul 18 01:21:45.395225 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:21:45.395180 2580 generic.go:356] "Generic (PLEG): container finished" podID="695f0f72-fba2-4a27-8c91-9db7b64b351b" containerID="74c91c460370adcad9eb5a08a90f5c4b34063a7f27efe49e81f1e821755cc690" exitCode=1 Jul 18 01:21:45.395693 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:21:45.395256 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" event={"ID":"695f0f72-fba2-4a27-8c91-9db7b64b351b","Type":"ContainerDied","Data":"74c91c460370adcad9eb5a08a90f5c4b34063a7f27efe49e81f1e821755cc690"} Jul 18 01:21:45.395693 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:21:45.395336 2580 scope.go:117] "RemoveContainer" containerID="ad7acb0c9b119f6cbd0fd7c8448d95ff0c21bac8e742b9fd623f708f2c0e4daf" Jul 18 01:21:45.395837 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:21:45.395762 2580 scope.go:117] "RemoveContainer" containerID="74c91c460370adcad9eb5a08a90f5c4b34063a7f27efe49e81f1e821755cc690" Jul 18 01:21:45.396011 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:21:45.395991 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w_odh-ai-gateway-infra(695f0f72-fba2-4a27-8c91-9db7b64b351b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" Jul 18 01:21:47.959870 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:21:47.959831 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" Jul 18 01:21:47.960357 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:21:47.960289 2580 scope.go:117] "RemoveContainer" containerID="74c91c460370adcad9eb5a08a90f5c4b34063a7f27efe49e81f1e821755cc690" Jul 18 01:21:47.960542 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:21:47.960519 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w_odh-ai-gateway-infra(695f0f72-fba2-4a27-8c91-9db7b64b351b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" Jul 18 01:21:51.615219 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:21:51.615132 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w"] Jul 18 01:21:51.755000 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:21:51.754969 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" Jul 18 01:21:51.807761 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:21:51.807727 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gc6jf\" (UniqueName: \"kubernetes.io/projected/695f0f72-fba2-4a27-8c91-9db7b64b351b-kube-api-access-gc6jf\") pod \"695f0f72-fba2-4a27-8c91-9db7b64b351b\" (UID: \"695f0f72-fba2-4a27-8c91-9db7b64b351b\") " Jul 18 01:21:51.807968 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:21:51.807791 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/695f0f72-fba2-4a27-8c91-9db7b64b351b-maas-api-tls\") pod \"695f0f72-fba2-4a27-8c91-9db7b64b351b\" (UID: \"695f0f72-fba2-4a27-8c91-9db7b64b351b\") " Jul 18 01:21:51.807968 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:21:51.807832 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/695f0f72-fba2-4a27-8c91-9db7b64b351b-openshift-service-ca\") pod \"695f0f72-fba2-4a27-8c91-9db7b64b351b\" (UID: \"695f0f72-fba2-4a27-8c91-9db7b64b351b\") " Jul 18 01:21:51.808282 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:21:51.808256 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/695f0f72-fba2-4a27-8c91-9db7b64b351b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "695f0f72-fba2-4a27-8c91-9db7b64b351b" (UID: "695f0f72-fba2-4a27-8c91-9db7b64b351b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 01:21:51.810103 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:21:51.810078 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/695f0f72-fba2-4a27-8c91-9db7b64b351b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "695f0f72-fba2-4a27-8c91-9db7b64b351b" (UID: "695f0f72-fba2-4a27-8c91-9db7b64b351b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 01:21:51.810103 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:21:51.810084 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695f0f72-fba2-4a27-8c91-9db7b64b351b-kube-api-access-gc6jf" (OuterVolumeSpecName: "kube-api-access-gc6jf") pod "695f0f72-fba2-4a27-8c91-9db7b64b351b" (UID: "695f0f72-fba2-4a27-8c91-9db7b64b351b"). InnerVolumeSpecName "kube-api-access-gc6jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 01:21:51.909525 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:21:51.909435 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc6jf\" (UniqueName: \"kubernetes.io/projected/695f0f72-fba2-4a27-8c91-9db7b64b351b-kube-api-access-gc6jf\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:21:51.909525 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:21:51.909468 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/695f0f72-fba2-4a27-8c91-9db7b64b351b-maas-api-tls\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:21:51.909525 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:21:51.909479 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/695f0f72-fba2-4a27-8c91-9db7b64b351b-openshift-service-ca\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:21:52.433038 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:21:52.433006 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" Jul 18 01:21:52.433038 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:21:52.433014 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w" event={"ID":"695f0f72-fba2-4a27-8c91-9db7b64b351b","Type":"ContainerDied","Data":"c7fde728e9313a93c7459346e14badda971dfba0067b86a19284c976271ea0de"} Jul 18 01:21:52.433289 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:21:52.433080 2580 scope.go:117] "RemoveContainer" containerID="74c91c460370adcad9eb5a08a90f5c4b34063a7f27efe49e81f1e821755cc690" Jul 18 01:21:52.452836 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:21:52.452806 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w"] Jul 18 01:21:52.454825 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:21:52.454805 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8942d9e2-85c5f7cfdd-ld84w"] Jul 18 01:21:54.149526 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:21:54.149489 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" path="/var/lib/kubelet/pods/695f0f72-fba2-4a27-8c91-9db7b64b351b/volumes" Jul 18 01:23:35.664471 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:35.663808 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz"] Jul 18 01:23:35.664471 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:35.664303 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" containerName="maas-api" Jul 18 01:23:35.664471 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:35.664350 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" containerName="maas-api" Jul 18 01:23:35.664471 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:35.664365 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" containerName="maas-api" Jul 18 01:23:35.664471 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:35.664375 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" containerName="maas-api" Jul 18 01:23:35.664471 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:35.664389 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" containerName="maas-api" Jul 18 01:23:35.664471 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:35.664398 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" containerName="maas-api" Jul 18 01:23:35.664471 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:35.664415 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" containerName="maas-api" Jul 18 01:23:35.664471 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:35.664422 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" containerName="maas-api" Jul 18 01:23:35.664471 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:35.664447 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" containerName="maas-api" Jul 18 01:23:35.664471 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:35.664458 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" containerName="maas-api" Jul 18 01:23:35.665522 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:35.664551 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" containerName="maas-api" Jul 18 01:23:35.665522 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:35.664562 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" containerName="maas-api" Jul 18 01:23:35.665522 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:35.664571 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" containerName="maas-api" Jul 18 01:23:35.665522 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:35.664581 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" containerName="maas-api" Jul 18 01:23:35.665522 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:35.664589 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" containerName="maas-api" Jul 18 01:23:35.665522 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:35.664600 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" containerName="maas-api" Jul 18 01:23:35.667906 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:35.667872 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz" Jul 18 01:23:35.670414 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:35.670390 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2djvc\"" Jul 18 01:23:35.670554 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:35.670412 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8d0c59be-serving-cert\"" Jul 18 01:23:35.676468 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:35.676410 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz"] Jul 18 01:23:35.735693 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:35.735637 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k69qg\" (UniqueName: \"kubernetes.io/projected/fb064ffb-4bf4-4838-ac76-fce2703f3907-kube-api-access-k69qg\") pod \"maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz\" (UID: \"fb064ffb-4bf4-4838-ac76-fce2703f3907\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz" Jul 18 01:23:35.735908 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:35.735787 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb064ffb-4bf4-4838-ac76-fce2703f3907-maas-api-tls\") pod \"maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz\" (UID: \"fb064ffb-4bf4-4838-ac76-fce2703f3907\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz" Jul 18 01:23:35.735908 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:35.735865 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb064ffb-4bf4-4838-ac76-fce2703f3907-openshift-service-ca\") pod \"maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz\" (UID: \"fb064ffb-4bf4-4838-ac76-fce2703f3907\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz" Jul 18 01:23:35.836451 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:35.836410 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb064ffb-4bf4-4838-ac76-fce2703f3907-openshift-service-ca\") pod \"maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz\" (UID: \"fb064ffb-4bf4-4838-ac76-fce2703f3907\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz" Jul 18 01:23:35.836663 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:35.836509 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k69qg\" (UniqueName: \"kubernetes.io/projected/fb064ffb-4bf4-4838-ac76-fce2703f3907-kube-api-access-k69qg\") pod \"maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz\" (UID: \"fb064ffb-4bf4-4838-ac76-fce2703f3907\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz" Jul 18 01:23:35.836663 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:35.836562 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb064ffb-4bf4-4838-ac76-fce2703f3907-maas-api-tls\") pod \"maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz\" (UID: \"fb064ffb-4bf4-4838-ac76-fce2703f3907\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz" Jul 18 01:23:35.837114 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:35.837092 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb064ffb-4bf4-4838-ac76-fce2703f3907-openshift-service-ca\") pod \"maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz\" (UID: \"fb064ffb-4bf4-4838-ac76-fce2703f3907\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz" Jul 18 01:23:35.839145 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:35.839122 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb064ffb-4bf4-4838-ac76-fce2703f3907-maas-api-tls\") pod \"maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz\" (UID: \"fb064ffb-4bf4-4838-ac76-fce2703f3907\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz" Jul 18 01:23:35.855349 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:35.855322 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k69qg\" (UniqueName: \"kubernetes.io/projected/fb064ffb-4bf4-4838-ac76-fce2703f3907-kube-api-access-k69qg\") pod \"maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz\" (UID: \"fb064ffb-4bf4-4838-ac76-fce2703f3907\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz" Jul 18 01:23:35.981044 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:35.981006 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz" Jul 18 01:23:36.122705 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:36.122657 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz"] Jul 18 01:23:36.978629 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:36.978582 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz" event={"ID":"fb064ffb-4bf4-4838-ac76-fce2703f3907","Type":"ContainerStarted","Data":"a615d37680e34c2fe458f5a365798eac3259306ecca2ec8ea047b24939d260d9"} Jul 18 01:23:36.979098 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:36.978692 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz" event={"ID":"fb064ffb-4bf4-4838-ac76-fce2703f3907","Type":"ContainerStarted","Data":"ad41c5a3ad891e48ce424c379c5e669f42007381304775f2920b64e8413bcd54"} Jul 18 01:23:36.979098 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:36.978805 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz" Jul 18 01:23:36.997020 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:36.996973 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz" podStartSLOduration=1.621011818 podStartE2EDuration="1.996958389s" totalImagesPullingTime="375.946571ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:23:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:23:36.995764304 +0000 UTC m=+2577.347614652" watchObservedRunningTime="2026-07-18 01:23:36.996958389 +0000 UTC m=+2577.348808734" Jul 18 01:23:42.988889 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:42.988857 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz" Jul 18 01:23:57.672202 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:57.672166 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz"] Jul 18 01:23:57.672646 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:57.672470 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz" podUID="fb064ffb-4bf4-4838-ac76-fce2703f3907" containerName="maas-api" containerID="cri-o://a615d37680e34c2fe458f5a365798eac3259306ecca2ec8ea047b24939d260d9" gracePeriod=30 Jul 18 01:23:57.924525 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:57.924456 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz" Jul 18 01:23:57.943426 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:57.943393 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb064ffb-4bf4-4838-ac76-fce2703f3907-maas-api-tls\") pod \"fb064ffb-4bf4-4838-ac76-fce2703f3907\" (UID: \"fb064ffb-4bf4-4838-ac76-fce2703f3907\") " Jul 18 01:23:57.943426 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:57.943434 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k69qg\" (UniqueName: \"kubernetes.io/projected/fb064ffb-4bf4-4838-ac76-fce2703f3907-kube-api-access-k69qg\") pod \"fb064ffb-4bf4-4838-ac76-fce2703f3907\" (UID: \"fb064ffb-4bf4-4838-ac76-fce2703f3907\") " Jul 18 01:23:57.943658 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:57.943525 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb064ffb-4bf4-4838-ac76-fce2703f3907-openshift-service-ca\") pod \"fb064ffb-4bf4-4838-ac76-fce2703f3907\" (UID: \"fb064ffb-4bf4-4838-ac76-fce2703f3907\") " Jul 18 01:23:57.943937 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:57.943909 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb064ffb-4bf4-4838-ac76-fce2703f3907-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fb064ffb-4bf4-4838-ac76-fce2703f3907" (UID: "fb064ffb-4bf4-4838-ac76-fce2703f3907"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 01:23:57.945865 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:57.945841 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb064ffb-4bf4-4838-ac76-fce2703f3907-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fb064ffb-4bf4-4838-ac76-fce2703f3907" (UID: "fb064ffb-4bf4-4838-ac76-fce2703f3907"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 01:23:57.945865 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:57.945847 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb064ffb-4bf4-4838-ac76-fce2703f3907-kube-api-access-k69qg" (OuterVolumeSpecName: "kube-api-access-k69qg") pod "fb064ffb-4bf4-4838-ac76-fce2703f3907" (UID: "fb064ffb-4bf4-4838-ac76-fce2703f3907"). InnerVolumeSpecName "kube-api-access-k69qg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 01:23:58.044463 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:58.044427 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb064ffb-4bf4-4838-ac76-fce2703f3907-openshift-service-ca\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:23:58.044463 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:58.044459 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb064ffb-4bf4-4838-ac76-fce2703f3907-maas-api-tls\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:23:58.044463 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:58.044471 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k69qg\" (UniqueName: \"kubernetes.io/projected/fb064ffb-4bf4-4838-ac76-fce2703f3907-kube-api-access-k69qg\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:23:58.096826 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:58.096791 2580 generic.go:356] "Generic (PLEG): container finished" podID="fb064ffb-4bf4-4838-ac76-fce2703f3907" containerID="a615d37680e34c2fe458f5a365798eac3259306ecca2ec8ea047b24939d260d9" exitCode=0 Jul 18 01:23:58.096962 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:58.096835 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz" Jul 18 01:23:58.096962 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:58.096881 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz" event={"ID":"fb064ffb-4bf4-4838-ac76-fce2703f3907","Type":"ContainerDied","Data":"a615d37680e34c2fe458f5a365798eac3259306ecca2ec8ea047b24939d260d9"} Jul 18 01:23:58.096962 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:58.096933 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz" event={"ID":"fb064ffb-4bf4-4838-ac76-fce2703f3907","Type":"ContainerDied","Data":"ad41c5a3ad891e48ce424c379c5e669f42007381304775f2920b64e8413bcd54"} Jul 18 01:23:58.096962 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:58.096957 2580 scope.go:117] "RemoveContainer" containerID="a615d37680e34c2fe458f5a365798eac3259306ecca2ec8ea047b24939d260d9" Jul 18 01:23:58.106244 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:58.106226 2580 scope.go:117] "RemoveContainer" containerID="a615d37680e34c2fe458f5a365798eac3259306ecca2ec8ea047b24939d260d9" Jul 18 01:23:58.106540 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:23:58.106521 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a615d37680e34c2fe458f5a365798eac3259306ecca2ec8ea047b24939d260d9\": container with ID starting with a615d37680e34c2fe458f5a365798eac3259306ecca2ec8ea047b24939d260d9 not found: ID does not exist" containerID="a615d37680e34c2fe458f5a365798eac3259306ecca2ec8ea047b24939d260d9" Jul 18 01:23:58.106599 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:58.106549 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a615d37680e34c2fe458f5a365798eac3259306ecca2ec8ea047b24939d260d9"} err="failed to get container status \"a615d37680e34c2fe458f5a365798eac3259306ecca2ec8ea047b24939d260d9\": rpc error: code = NotFound desc = could not find container \"a615d37680e34c2fe458f5a365798eac3259306ecca2ec8ea047b24939d260d9\": container with ID starting with a615d37680e34c2fe458f5a365798eac3259306ecca2ec8ea047b24939d260d9 not found: ID does not exist" Jul 18 01:23:58.118103 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:58.118075 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz"] Jul 18 01:23:58.120124 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:58.120103 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8d0c59be-85bf54c869-xpnhz"] Jul 18 01:23:58.149969 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:23:58.149940 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb064ffb-4bf4-4838-ac76-fce2703f3907" path="/var/lib/kubelet/pods/fb064ffb-4bf4-4838-ac76-fce2703f3907/volumes" Jul 18 01:24:35.086226 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:24:35.086188 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn"] Jul 18 01:24:35.086847 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:24:35.086721 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" containerName="maas-api" Jul 18 01:24:35.086847 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:24:35.086742 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" containerName="maas-api" Jul 18 01:24:35.086847 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:24:35.086778 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" containerName="maas-api" Jul 18 01:24:35.086847 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:24:35.086787 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef8f2576-4655-4e94-bcf1-2a9e4fba7ece" containerName="maas-api" Jul 18 01:24:35.086847 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:24:35.086797 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb064ffb-4bf4-4838-ac76-fce2703f3907" containerName="maas-api" Jul 18 01:24:35.086847 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:24:35.086806 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb064ffb-4bf4-4838-ac76-fce2703f3907" containerName="maas-api" Jul 18 01:24:35.087164 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:24:35.086895 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb064ffb-4bf4-4838-ac76-fce2703f3907" containerName="maas-api" Jul 18 01:24:35.087164 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:24:35.086910 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" containerName="maas-api" Jul 18 01:24:35.090032 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:24:35.090007 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn" Jul 18 01:24:35.092883 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:24:35.092855 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f92ba3bc-serving-cert\"" Jul 18 01:24:35.093003 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:24:35.092905 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2djvc\"" Jul 18 01:24:35.112875 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:24:35.112843 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn"] Jul 18 01:24:35.191396 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:24:35.191358 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/383a7339-aec0-4114-b8a5-ed12cea2ce35-openshift-service-ca\") pod \"maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn\" (UID: \"383a7339-aec0-4114-b8a5-ed12cea2ce35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn" Jul 18 01:24:35.191396 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:24:35.191399 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44z5b\" (UniqueName: \"kubernetes.io/projected/383a7339-aec0-4114-b8a5-ed12cea2ce35-kube-api-access-44z5b\") pod \"maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn\" (UID: \"383a7339-aec0-4114-b8a5-ed12cea2ce35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn" Jul 18 01:24:35.191589 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:24:35.191482 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/383a7339-aec0-4114-b8a5-ed12cea2ce35-maas-api-tls\") pod \"maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn\" (UID: \"383a7339-aec0-4114-b8a5-ed12cea2ce35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn" Jul 18 01:24:35.291968 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:24:35.291936 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/383a7339-aec0-4114-b8a5-ed12cea2ce35-openshift-service-ca\") pod \"maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn\" (UID: \"383a7339-aec0-4114-b8a5-ed12cea2ce35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn" Jul 18 01:24:35.291968 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:24:35.291973 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44z5b\" (UniqueName: \"kubernetes.io/projected/383a7339-aec0-4114-b8a5-ed12cea2ce35-kube-api-access-44z5b\") pod \"maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn\" (UID: \"383a7339-aec0-4114-b8a5-ed12cea2ce35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn" Jul 18 01:24:35.292183 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:24:35.292025 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/383a7339-aec0-4114-b8a5-ed12cea2ce35-maas-api-tls\") pod \"maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn\" (UID: \"383a7339-aec0-4114-b8a5-ed12cea2ce35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn" Jul 18 01:24:35.292183 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:24:35.292118 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f92ba3bc-serving-cert: secret "maas-api-e2e-mt-f92ba3bc-serving-cert" not found Jul 18 01:24:35.292266 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:24:35.292192 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/383a7339-aec0-4114-b8a5-ed12cea2ce35-maas-api-tls podName:383a7339-aec0-4114-b8a5-ed12cea2ce35 nodeName:}" failed. No retries permitted until 2026-07-18 01:24:35.792172044 +0000 UTC m=+2636.144022387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/383a7339-aec0-4114-b8a5-ed12cea2ce35-maas-api-tls") pod "maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn" (UID: "383a7339-aec0-4114-b8a5-ed12cea2ce35") : secret "maas-api-e2e-mt-f92ba3bc-serving-cert" not found Jul 18 01:24:35.292603 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:24:35.292581 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/383a7339-aec0-4114-b8a5-ed12cea2ce35-openshift-service-ca\") pod \"maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn\" (UID: \"383a7339-aec0-4114-b8a5-ed12cea2ce35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn" Jul 18 01:24:35.300934 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:24:35.300902 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44z5b\" (UniqueName: \"kubernetes.io/projected/383a7339-aec0-4114-b8a5-ed12cea2ce35-kube-api-access-44z5b\") pod \"maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn\" (UID: \"383a7339-aec0-4114-b8a5-ed12cea2ce35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn" Jul 18 01:24:35.796634 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:24:35.796595 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/383a7339-aec0-4114-b8a5-ed12cea2ce35-maas-api-tls\") pod \"maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn\" (UID: \"383a7339-aec0-4114-b8a5-ed12cea2ce35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn" Jul 18 01:24:35.799212 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:24:35.799186 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/383a7339-aec0-4114-b8a5-ed12cea2ce35-maas-api-tls\") pod \"maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn\" (UID: \"383a7339-aec0-4114-b8a5-ed12cea2ce35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn" Jul 18 01:24:36.005816 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:24:36.005783 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn" Jul 18 01:24:36.148711 ip-10-0-134-239 kubenswrapper[2580]: W0718 01:24:36.148662 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod383a7339_aec0_4114_b8a5_ed12cea2ce35.slice/crio-bff7d758def4d27ff7ea012254d0b3a9f7981106efca5715333cd97d9ccfc7a4 WatchSource:0}: Error finding container bff7d758def4d27ff7ea012254d0b3a9f7981106efca5715333cd97d9ccfc7a4: Status 404 returned error can't find the container with id bff7d758def4d27ff7ea012254d0b3a9f7981106efca5715333cd97d9ccfc7a4 Jul 18 01:24:36.150063 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:24:36.150041 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 01:24:36.150645 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:24:36.150625 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn"] Jul 18 01:24:36.295081 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:24:36.295042 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn" event={"ID":"383a7339-aec0-4114-b8a5-ed12cea2ce35","Type":"ContainerStarted","Data":"bff7d758def4d27ff7ea012254d0b3a9f7981106efca5715333cd97d9ccfc7a4"} Jul 18 01:24:37.302130 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:24:37.302088 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn" event={"ID":"383a7339-aec0-4114-b8a5-ed12cea2ce35","Type":"ContainerStarted","Data":"b1fb1e7a2b6b2305497204b44a937d14ef59b90e338d9cb7c96b974d0f3bad44"} Jul 18 01:24:37.302536 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:24:37.302155 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn" Jul 18 01:24:37.321349 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:24:37.321297 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn" podStartSLOduration=1.8712533489999998 podStartE2EDuration="2.321281112s" totalImagesPullingTime="450.027763ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:24:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:24:37.31956712 +0000 UTC m=+2637.671417457" watchObservedRunningTime="2026-07-18 01:24:37.321281112 +0000 UTC m=+2637.673131457" Jul 18 01:24:43.312804 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:24:43.312777 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn" Jul 18 01:25:38.189897 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:25:38.189642 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn"] Jul 18 01:25:38.190682 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:25:38.189924 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn" podUID="383a7339-aec0-4114-b8a5-ed12cea2ce35" containerName="maas-api" containerID="cri-o://b1fb1e7a2b6b2305497204b44a937d14ef59b90e338d9cb7c96b974d0f3bad44" gracePeriod=30 Jul 18 01:25:38.308873 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:25:38.308826 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn" podUID="383a7339-aec0-4114-b8a5-ed12cea2ce35" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.49:8443/health\": dial tcp 10.134.0.49:8443: connect: connection refused" Jul 18 01:25:38.437171 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:25:38.437146 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn" Jul 18 01:25:38.589995 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:25:38.589961 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44z5b\" (UniqueName: \"kubernetes.io/projected/383a7339-aec0-4114-b8a5-ed12cea2ce35-kube-api-access-44z5b\") pod \"383a7339-aec0-4114-b8a5-ed12cea2ce35\" (UID: \"383a7339-aec0-4114-b8a5-ed12cea2ce35\") " Jul 18 01:25:38.590178 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:25:38.590015 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/383a7339-aec0-4114-b8a5-ed12cea2ce35-maas-api-tls\") pod \"383a7339-aec0-4114-b8a5-ed12cea2ce35\" (UID: \"383a7339-aec0-4114-b8a5-ed12cea2ce35\") " Jul 18 01:25:38.590178 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:25:38.590048 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/383a7339-aec0-4114-b8a5-ed12cea2ce35-openshift-service-ca\") pod \"383a7339-aec0-4114-b8a5-ed12cea2ce35\" (UID: \"383a7339-aec0-4114-b8a5-ed12cea2ce35\") " Jul 18 01:25:38.590448 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:25:38.590425 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/383a7339-aec0-4114-b8a5-ed12cea2ce35-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "383a7339-aec0-4114-b8a5-ed12cea2ce35" (UID: "383a7339-aec0-4114-b8a5-ed12cea2ce35"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 01:25:38.592357 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:25:38.592332 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/383a7339-aec0-4114-b8a5-ed12cea2ce35-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "383a7339-aec0-4114-b8a5-ed12cea2ce35" (UID: "383a7339-aec0-4114-b8a5-ed12cea2ce35"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 01:25:38.592357 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:25:38.592338 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/383a7339-aec0-4114-b8a5-ed12cea2ce35-kube-api-access-44z5b" (OuterVolumeSpecName: "kube-api-access-44z5b") pod "383a7339-aec0-4114-b8a5-ed12cea2ce35" (UID: "383a7339-aec0-4114-b8a5-ed12cea2ce35"). InnerVolumeSpecName "kube-api-access-44z5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 01:25:38.620918 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:25:38.620884 2580 generic.go:356] "Generic (PLEG): container finished" podID="383a7339-aec0-4114-b8a5-ed12cea2ce35" containerID="b1fb1e7a2b6b2305497204b44a937d14ef59b90e338d9cb7c96b974d0f3bad44" exitCode=0 Jul 18 01:25:38.621081 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:25:38.620928 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn" Jul 18 01:25:38.621081 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:25:38.620943 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn" event={"ID":"383a7339-aec0-4114-b8a5-ed12cea2ce35","Type":"ContainerDied","Data":"b1fb1e7a2b6b2305497204b44a937d14ef59b90e338d9cb7c96b974d0f3bad44"} Jul 18 01:25:38.621081 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:25:38.620997 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn" event={"ID":"383a7339-aec0-4114-b8a5-ed12cea2ce35","Type":"ContainerDied","Data":"bff7d758def4d27ff7ea012254d0b3a9f7981106efca5715333cd97d9ccfc7a4"} Jul 18 01:25:38.621081 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:25:38.621019 2580 scope.go:117] "RemoveContainer" containerID="b1fb1e7a2b6b2305497204b44a937d14ef59b90e338d9cb7c96b974d0f3bad44" Jul 18 01:25:38.630241 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:25:38.630223 2580 scope.go:117] "RemoveContainer" containerID="b1fb1e7a2b6b2305497204b44a937d14ef59b90e338d9cb7c96b974d0f3bad44" Jul 18 01:25:38.630550 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:25:38.630529 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1fb1e7a2b6b2305497204b44a937d14ef59b90e338d9cb7c96b974d0f3bad44\": container with ID starting with b1fb1e7a2b6b2305497204b44a937d14ef59b90e338d9cb7c96b974d0f3bad44 not found: ID does not exist" containerID="b1fb1e7a2b6b2305497204b44a937d14ef59b90e338d9cb7c96b974d0f3bad44" Jul 18 01:25:38.630606 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:25:38.630560 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1fb1e7a2b6b2305497204b44a937d14ef59b90e338d9cb7c96b974d0f3bad44"} err="failed to get container status \"b1fb1e7a2b6b2305497204b44a937d14ef59b90e338d9cb7c96b974d0f3bad44\": rpc error: code = NotFound desc = could not find container \"b1fb1e7a2b6b2305497204b44a937d14ef59b90e338d9cb7c96b974d0f3bad44\": container with ID starting with b1fb1e7a2b6b2305497204b44a937d14ef59b90e338d9cb7c96b974d0f3bad44 not found: ID does not exist" Jul 18 01:25:38.643782 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:25:38.643752 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn"] Jul 18 01:25:38.647756 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:25:38.647729 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f92ba3bc-7fd6c7674f-vzbtn"] Jul 18 01:25:38.691767 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:25:38.691728 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44z5b\" (UniqueName: \"kubernetes.io/projected/383a7339-aec0-4114-b8a5-ed12cea2ce35-kube-api-access-44z5b\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:25:38.691767 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:25:38.691759 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/383a7339-aec0-4114-b8a5-ed12cea2ce35-maas-api-tls\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:25:38.691767 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:25:38.691769 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/383a7339-aec0-4114-b8a5-ed12cea2ce35-openshift-service-ca\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:25:40.149752 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:25:40.149718 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="383a7339-aec0-4114-b8a5-ed12cea2ce35" path="/var/lib/kubelet/pods/383a7339-aec0-4114-b8a5-ed12cea2ce35/volumes" Jul 18 01:27:42.281107 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:42.281065 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v"] Jul 18 01:27:42.281758 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:42.281589 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="383a7339-aec0-4114-b8a5-ed12cea2ce35" containerName="maas-api" Jul 18 01:27:42.281758 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:42.281606 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="383a7339-aec0-4114-b8a5-ed12cea2ce35" containerName="maas-api" Jul 18 01:27:42.281758 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:42.281619 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" containerName="maas-api" Jul 18 01:27:42.281758 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:42.281629 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="695f0f72-fba2-4a27-8c91-9db7b64b351b" containerName="maas-api" Jul 18 01:27:42.281758 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:42.281733 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="383a7339-aec0-4114-b8a5-ed12cea2ce35" containerName="maas-api" Jul 18 01:27:42.285203 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:42.285179 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v" Jul 18 01:27:42.288475 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:42.288451 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2djvc\"" Jul 18 01:27:42.290411 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:42.290384 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1e72d0-serving-cert\"" Jul 18 01:27:42.300085 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:42.300056 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v"] Jul 18 01:27:42.417504 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:42.417463 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bec093a-dc5c-46df-8db1-4e6f46a201be-maas-api-tls\") pod \"maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v\" (UID: \"4bec093a-dc5c-46df-8db1-4e6f46a201be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v" Jul 18 01:27:42.417716 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:42.417510 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bec093a-dc5c-46df-8db1-4e6f46a201be-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v\" (UID: \"4bec093a-dc5c-46df-8db1-4e6f46a201be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v" Jul 18 01:27:42.417716 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:42.417599 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcgjc\" (UniqueName: \"kubernetes.io/projected/4bec093a-dc5c-46df-8db1-4e6f46a201be-kube-api-access-hcgjc\") pod \"maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v\" (UID: \"4bec093a-dc5c-46df-8db1-4e6f46a201be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v" Jul 18 01:27:42.519209 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:42.519168 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcgjc\" (UniqueName: \"kubernetes.io/projected/4bec093a-dc5c-46df-8db1-4e6f46a201be-kube-api-access-hcgjc\") pod \"maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v\" (UID: \"4bec093a-dc5c-46df-8db1-4e6f46a201be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v" Jul 18 01:27:42.519411 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:42.519296 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bec093a-dc5c-46df-8db1-4e6f46a201be-maas-api-tls\") pod \"maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v\" (UID: \"4bec093a-dc5c-46df-8db1-4e6f46a201be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v" Jul 18 01:27:42.519411 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:42.519339 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bec093a-dc5c-46df-8db1-4e6f46a201be-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v\" (UID: \"4bec093a-dc5c-46df-8db1-4e6f46a201be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v" Jul 18 01:27:42.519725 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:27:42.519694 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-1e72d0-serving-cert: secret "maas-api-e2e-inf-a-1e72d0-serving-cert" not found Jul 18 01:27:42.519859 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:27:42.519761 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bec093a-dc5c-46df-8db1-4e6f46a201be-maas-api-tls podName:4bec093a-dc5c-46df-8db1-4e6f46a201be nodeName:}" failed. No retries permitted until 2026-07-18 01:27:43.019745382 +0000 UTC m=+2823.371595705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4bec093a-dc5c-46df-8db1-4e6f46a201be-maas-api-tls") pod "maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v" (UID: "4bec093a-dc5c-46df-8db1-4e6f46a201be") : secret "maas-api-e2e-inf-a-1e72d0-serving-cert" not found Jul 18 01:27:42.520137 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:42.520119 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bec093a-dc5c-46df-8db1-4e6f46a201be-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v\" (UID: \"4bec093a-dc5c-46df-8db1-4e6f46a201be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v" Jul 18 01:27:42.528890 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:42.528867 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcgjc\" (UniqueName: \"kubernetes.io/projected/4bec093a-dc5c-46df-8db1-4e6f46a201be-kube-api-access-hcgjc\") pod \"maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v\" (UID: \"4bec093a-dc5c-46df-8db1-4e6f46a201be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v" Jul 18 01:27:43.025824 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:43.025777 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bec093a-dc5c-46df-8db1-4e6f46a201be-maas-api-tls\") pod \"maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v\" (UID: \"4bec093a-dc5c-46df-8db1-4e6f46a201be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v" Jul 18 01:27:43.028373 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:43.028353 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bec093a-dc5c-46df-8db1-4e6f46a201be-maas-api-tls\") pod \"maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v\" (UID: \"4bec093a-dc5c-46df-8db1-4e6f46a201be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v" Jul 18 01:27:43.196409 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:43.196366 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v" Jul 18 01:27:43.341232 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:43.341191 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v"] Jul 18 01:27:43.342761 ip-10-0-134-239 kubenswrapper[2580]: W0718 01:27:43.342729 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bec093a_dc5c_46df_8db1_4e6f46a201be.slice/crio-3fe5521cdc1d97174569fd599ab4e63cdc9749fa9e3b009ec4f141c7946d569b WatchSource:0}: Error finding container 3fe5521cdc1d97174569fd599ab4e63cdc9749fa9e3b009ec4f141c7946d569b: Status 404 returned error can't find the container with id 3fe5521cdc1d97174569fd599ab4e63cdc9749fa9e3b009ec4f141c7946d569b Jul 18 01:27:44.263220 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:44.263184 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v" event={"ID":"4bec093a-dc5c-46df-8db1-4e6f46a201be","Type":"ContainerStarted","Data":"cee49eb7d3241f95def8543e8f96d02acadacc8e46feefe9157c26190cd19309"} Jul 18 01:27:44.263405 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:44.263240 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v" event={"ID":"4bec093a-dc5c-46df-8db1-4e6f46a201be","Type":"ContainerStarted","Data":"3fe5521cdc1d97174569fd599ab4e63cdc9749fa9e3b009ec4f141c7946d569b"} Jul 18 01:27:44.263405 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:44.263291 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v" Jul 18 01:27:50.274897 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:50.274867 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v" Jul 18 01:27:50.295748 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:50.295687 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v" podStartSLOduration=7.823470147 podStartE2EDuration="8.295657393s" totalImagesPullingTime="472.187246ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:27:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:27:44.27938288 +0000 UTC m=+2824.631233228" watchObservedRunningTime="2026-07-18 01:27:50.295657393 +0000 UTC m=+2830.647507738" Jul 18 01:27:50.313215 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:50.313180 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj"] Jul 18 01:27:50.316934 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:50.316912 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj" Jul 18 01:27:50.319687 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:50.319639 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-8e3c6c-serving-cert\"" Jul 18 01:27:50.331921 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:50.331889 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj"] Jul 18 01:27:50.396969 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:50.396929 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tjgq\" (UniqueName: \"kubernetes.io/projected/f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b-kube-api-access-2tjgq\") pod \"maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj\" (UID: \"f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj" Jul 18 01:27:50.397170 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:50.396990 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b-maas-api-tls\") pod \"maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj\" (UID: \"f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj" Jul 18 01:27:50.397170 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:50.397096 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj\" (UID: \"f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj" Jul 18 01:27:50.498521 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:50.498490 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj\" (UID: \"f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj" Jul 18 01:27:50.498722 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:50.498581 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tjgq\" (UniqueName: \"kubernetes.io/projected/f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b-kube-api-access-2tjgq\") pod \"maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj\" (UID: \"f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj" Jul 18 01:27:50.498722 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:50.498611 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b-maas-api-tls\") pod \"maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj\" (UID: \"f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj" Jul 18 01:27:50.498814 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:27:50.498749 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-8e3c6c-serving-cert: secret "maas-api-e2e-inf-b-8e3c6c-serving-cert" not found Jul 18 01:27:50.498858 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:27:50.498816 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b-maas-api-tls podName:f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b nodeName:}" failed. No retries permitted until 2026-07-18 01:27:50.998797001 +0000 UTC m=+2831.350647326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b-maas-api-tls") pod "maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj" (UID: "f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b") : secret "maas-api-e2e-inf-b-8e3c6c-serving-cert" not found Jul 18 01:27:50.499315 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:50.499294 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj\" (UID: \"f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj" Jul 18 01:27:50.519397 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:50.519361 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tjgq\" (UniqueName: \"kubernetes.io/projected/f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b-kube-api-access-2tjgq\") pod \"maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj\" (UID: \"f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj" Jul 18 01:27:51.002329 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:51.002281 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b-maas-api-tls\") pod \"maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj\" (UID: \"f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj" Jul 18 01:27:51.004999 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:51.004966 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b-maas-api-tls\") pod \"maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj\" (UID: \"f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj" Jul 18 01:27:51.230890 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:51.230850 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj" Jul 18 01:27:51.383369 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:51.377913 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj"] Jul 18 01:27:52.308740 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:52.308707 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj" event={"ID":"f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b","Type":"ContainerStarted","Data":"2eeaa6042fa16fea8e1db9438cd23a7b41579805b6b8bef8f75cce76f100bf8f"} Jul 18 01:27:52.308930 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:52.308761 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj" event={"ID":"f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b","Type":"ContainerStarted","Data":"1f5ef0ee24a720cde9a84aeb3137badfb35ddcba775e373a65b9df17055eb53e"} Jul 18 01:27:52.308930 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:52.308830 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj" Jul 18 01:27:52.327607 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:52.327558 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj" podStartSLOduration=1.7598775610000001 podStartE2EDuration="2.327542654s" totalImagesPullingTime="567.665093ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:27:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:27:52.326039569 +0000 UTC m=+2832.677889917" watchObservedRunningTime="2026-07-18 01:27:52.327542654 +0000 UTC m=+2832.679392999" Jul 18 01:27:58.319198 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:27:58.319168 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj" Jul 18 01:29:09.529465 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:09.529431 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v"] Jul 18 01:29:09.529932 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:09.529693 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v" podUID="4bec093a-dc5c-46df-8db1-4e6f46a201be" containerName="maas-api" containerID="cri-o://cee49eb7d3241f95def8543e8f96d02acadacc8e46feefe9157c26190cd19309" gracePeriod=30 Jul 18 01:29:09.712731 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:09.712662 2580 generic.go:356] "Generic (PLEG): container finished" podID="4bec093a-dc5c-46df-8db1-4e6f46a201be" containerID="cee49eb7d3241f95def8543e8f96d02acadacc8e46feefe9157c26190cd19309" exitCode=0 Jul 18 01:29:09.712731 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:09.712711 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v" event={"ID":"4bec093a-dc5c-46df-8db1-4e6f46a201be","Type":"ContainerDied","Data":"cee49eb7d3241f95def8543e8f96d02acadacc8e46feefe9157c26190cd19309"} Jul 18 01:29:09.789664 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:09.789599 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v" Jul 18 01:29:09.925392 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:09.925349 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bec093a-dc5c-46df-8db1-4e6f46a201be-openshift-service-ca\") pod \"4bec093a-dc5c-46df-8db1-4e6f46a201be\" (UID: \"4bec093a-dc5c-46df-8db1-4e6f46a201be\") " Jul 18 01:29:09.925609 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:09.925453 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcgjc\" (UniqueName: \"kubernetes.io/projected/4bec093a-dc5c-46df-8db1-4e6f46a201be-kube-api-access-hcgjc\") pod \"4bec093a-dc5c-46df-8db1-4e6f46a201be\" (UID: \"4bec093a-dc5c-46df-8db1-4e6f46a201be\") " Jul 18 01:29:09.925712 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:09.925611 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bec093a-dc5c-46df-8db1-4e6f46a201be-maas-api-tls\") pod \"4bec093a-dc5c-46df-8db1-4e6f46a201be\" (UID: \"4bec093a-dc5c-46df-8db1-4e6f46a201be\") " Jul 18 01:29:09.925862 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:09.925837 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bec093a-dc5c-46df-8db1-4e6f46a201be-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4bec093a-dc5c-46df-8db1-4e6f46a201be" (UID: "4bec093a-dc5c-46df-8db1-4e6f46a201be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 01:29:09.925988 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:09.925972 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bec093a-dc5c-46df-8db1-4e6f46a201be-openshift-service-ca\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:29:09.927801 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:09.927779 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bec093a-dc5c-46df-8db1-4e6f46a201be-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4bec093a-dc5c-46df-8db1-4e6f46a201be" (UID: "4bec093a-dc5c-46df-8db1-4e6f46a201be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 01:29:09.927874 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:09.927799 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bec093a-dc5c-46df-8db1-4e6f46a201be-kube-api-access-hcgjc" (OuterVolumeSpecName: "kube-api-access-hcgjc") pod "4bec093a-dc5c-46df-8db1-4e6f46a201be" (UID: "4bec093a-dc5c-46df-8db1-4e6f46a201be"). InnerVolumeSpecName "kube-api-access-hcgjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 01:29:10.026612 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:10.026576 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcgjc\" (UniqueName: \"kubernetes.io/projected/4bec093a-dc5c-46df-8db1-4e6f46a201be-kube-api-access-hcgjc\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:29:10.026612 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:10.026606 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bec093a-dc5c-46df-8db1-4e6f46a201be-maas-api-tls\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:29:10.718468 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:10.718434 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v" event={"ID":"4bec093a-dc5c-46df-8db1-4e6f46a201be","Type":"ContainerDied","Data":"3fe5521cdc1d97174569fd599ab4e63cdc9749fa9e3b009ec4f141c7946d569b"} Jul 18 01:29:10.718468 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:10.718454 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v" Jul 18 01:29:10.718976 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:10.718493 2580 scope.go:117] "RemoveContainer" containerID="cee49eb7d3241f95def8543e8f96d02acadacc8e46feefe9157c26190cd19309" Jul 18 01:29:10.734730 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:10.734701 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v"] Jul 18 01:29:10.738099 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:10.738077 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1e72d0-59f96b4b58-jp29v"] Jul 18 01:29:12.154625 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:12.154592 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bec093a-dc5c-46df-8db1-4e6f46a201be" path="/var/lib/kubelet/pods/4bec093a-dc5c-46df-8db1-4e6f46a201be/volumes" Jul 18 01:29:35.651032 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:35.650973 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj"] Jul 18 01:29:35.651611 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:35.651238 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj" podUID="f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b" containerName="maas-api" containerID="cri-o://2eeaa6042fa16fea8e1db9438cd23a7b41579805b6b8bef8f75cce76f100bf8f" gracePeriod=30 Jul 18 01:29:35.847355 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:35.847324 2580 generic.go:356] "Generic (PLEG): container finished" podID="f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b" containerID="2eeaa6042fa16fea8e1db9438cd23a7b41579805b6b8bef8f75cce76f100bf8f" exitCode=0 Jul 18 01:29:35.847518 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:35.847407 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj" event={"ID":"f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b","Type":"ContainerDied","Data":"2eeaa6042fa16fea8e1db9438cd23a7b41579805b6b8bef8f75cce76f100bf8f"} Jul 18 01:29:35.910145 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:35.910066 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj" Jul 18 01:29:35.960195 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:35.960162 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tjgq\" (UniqueName: \"kubernetes.io/projected/f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b-kube-api-access-2tjgq\") pod \"f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b\" (UID: \"f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b\") " Jul 18 01:29:35.960365 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:35.960212 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b-openshift-service-ca\") pod \"f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b\" (UID: \"f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b\") " Jul 18 01:29:35.960365 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:35.960250 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b-maas-api-tls\") pod \"f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b\" (UID: \"f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b\") " Jul 18 01:29:35.960546 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:35.960525 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b" (UID: "f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 01:29:35.962404 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:35.962376 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b" (UID: "f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 01:29:35.962521 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:35.962451 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b-kube-api-access-2tjgq" (OuterVolumeSpecName: "kube-api-access-2tjgq") pod "f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b" (UID: "f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b"). InnerVolumeSpecName "kube-api-access-2tjgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 01:29:36.061700 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:36.061639 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b-maas-api-tls\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:29:36.061700 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:36.061695 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tjgq\" (UniqueName: \"kubernetes.io/projected/f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b-kube-api-access-2tjgq\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:29:36.061700 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:36.061705 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b-openshift-service-ca\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:29:36.853688 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:36.853636 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj" Jul 18 01:29:36.853688 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:36.853642 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj" event={"ID":"f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b","Type":"ContainerDied","Data":"1f5ef0ee24a720cde9a84aeb3137badfb35ddcba775e373a65b9df17055eb53e"} Jul 18 01:29:36.854231 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:36.853730 2580 scope.go:117] "RemoveContainer" containerID="2eeaa6042fa16fea8e1db9438cd23a7b41579805b6b8bef8f75cce76f100bf8f" Jul 18 01:29:36.871020 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:36.870989 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj"] Jul 18 01:29:36.876652 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:36.876622 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8e3c6c-fb6f5b676-4sxbj"] Jul 18 01:29:38.149255 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:38.149220 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b" path="/var/lib/kubelet/pods/f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b/volumes" Jul 18 01:29:55.403027 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:55.402989 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-feded4-5946cb944b-qnvdv"] Jul 18 01:29:55.403477 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:55.403463 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bec093a-dc5c-46df-8db1-4e6f46a201be" containerName="maas-api" Jul 18 01:29:55.403522 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:55.403481 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bec093a-dc5c-46df-8db1-4e6f46a201be" containerName="maas-api" Jul 18 01:29:55.403522 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:55.403516 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b" containerName="maas-api" Jul 18 01:29:55.403599 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:55.403524 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b" containerName="maas-api" Jul 18 01:29:55.403633 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:55.403598 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2e1b6a7-0642-4a42-9da7-31ce6ffb7c6b" containerName="maas-api" Jul 18 01:29:55.403633 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:55.403613 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bec093a-dc5c-46df-8db1-4e6f46a201be" containerName="maas-api" Jul 18 01:29:55.406826 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:55.406805 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-feded4-5946cb944b-qnvdv" Jul 18 01:29:55.409520 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:55.409494 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-feded4-serving-cert\"" Jul 18 01:29:55.409649 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:55.409518 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2djvc\"" Jul 18 01:29:55.421172 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:55.421148 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-feded4-5946cb944b-qnvdv"] Jul 18 01:29:55.536583 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:55.536546 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clkg9\" (UniqueName: \"kubernetes.io/projected/b52ae522-c264-467f-85c5-bc0ff444dbf8-kube-api-access-clkg9\") pod \"maas-api-e2e-api-a-feded4-5946cb944b-qnvdv\" (UID: \"b52ae522-c264-467f-85c5-bc0ff444dbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-feded4-5946cb944b-qnvdv" Jul 18 01:29:55.536809 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:55.536610 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b52ae522-c264-467f-85c5-bc0ff444dbf8-maas-api-tls\") pod \"maas-api-e2e-api-a-feded4-5946cb944b-qnvdv\" (UID: \"b52ae522-c264-467f-85c5-bc0ff444dbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-feded4-5946cb944b-qnvdv" Jul 18 01:29:55.536809 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:55.536714 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b52ae522-c264-467f-85c5-bc0ff444dbf8-openshift-service-ca\") pod \"maas-api-e2e-api-a-feded4-5946cb944b-qnvdv\" (UID: \"b52ae522-c264-467f-85c5-bc0ff444dbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-feded4-5946cb944b-qnvdv" Jul 18 01:29:55.637850 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:55.637812 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clkg9\" (UniqueName: \"kubernetes.io/projected/b52ae522-c264-467f-85c5-bc0ff444dbf8-kube-api-access-clkg9\") pod \"maas-api-e2e-api-a-feded4-5946cb944b-qnvdv\" (UID: \"b52ae522-c264-467f-85c5-bc0ff444dbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-feded4-5946cb944b-qnvdv" Jul 18 01:29:55.638041 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:55.637869 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b52ae522-c264-467f-85c5-bc0ff444dbf8-maas-api-tls\") pod \"maas-api-e2e-api-a-feded4-5946cb944b-qnvdv\" (UID: \"b52ae522-c264-467f-85c5-bc0ff444dbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-feded4-5946cb944b-qnvdv" Jul 18 01:29:55.638041 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:55.637897 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b52ae522-c264-467f-85c5-bc0ff444dbf8-openshift-service-ca\") pod \"maas-api-e2e-api-a-feded4-5946cb944b-qnvdv\" (UID: \"b52ae522-c264-467f-85c5-bc0ff444dbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-feded4-5946cb944b-qnvdv" Jul 18 01:29:55.638136 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:29:55.638043 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-feded4-serving-cert: secret "maas-api-e2e-api-a-feded4-serving-cert" not found Jul 18 01:29:55.638136 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:29:55.638130 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b52ae522-c264-467f-85c5-bc0ff444dbf8-maas-api-tls podName:b52ae522-c264-467f-85c5-bc0ff444dbf8 nodeName:}" failed. No retries permitted until 2026-07-18 01:29:56.138109414 +0000 UTC m=+2956.489959742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b52ae522-c264-467f-85c5-bc0ff444dbf8-maas-api-tls") pod "maas-api-e2e-api-a-feded4-5946cb944b-qnvdv" (UID: "b52ae522-c264-467f-85c5-bc0ff444dbf8") : secret "maas-api-e2e-api-a-feded4-serving-cert" not found Jul 18 01:29:55.638526 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:55.638509 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b52ae522-c264-467f-85c5-bc0ff444dbf8-openshift-service-ca\") pod \"maas-api-e2e-api-a-feded4-5946cb944b-qnvdv\" (UID: \"b52ae522-c264-467f-85c5-bc0ff444dbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-feded4-5946cb944b-qnvdv" Jul 18 01:29:55.647580 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:55.647544 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clkg9\" (UniqueName: \"kubernetes.io/projected/b52ae522-c264-467f-85c5-bc0ff444dbf8-kube-api-access-clkg9\") pod \"maas-api-e2e-api-a-feded4-5946cb944b-qnvdv\" (UID: \"b52ae522-c264-467f-85c5-bc0ff444dbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-feded4-5946cb944b-qnvdv" Jul 18 01:29:56.142964 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:56.142921 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b52ae522-c264-467f-85c5-bc0ff444dbf8-maas-api-tls\") pod \"maas-api-e2e-api-a-feded4-5946cb944b-qnvdv\" (UID: \"b52ae522-c264-467f-85c5-bc0ff444dbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-feded4-5946cb944b-qnvdv" Jul 18 01:29:56.145638 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:56.145616 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b52ae522-c264-467f-85c5-bc0ff444dbf8-maas-api-tls\") pod \"maas-api-e2e-api-a-feded4-5946cb944b-qnvdv\" (UID: \"b52ae522-c264-467f-85c5-bc0ff444dbf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-feded4-5946cb944b-qnvdv" Jul 18 01:29:56.320782 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:56.320737 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-feded4-5946cb944b-qnvdv" Jul 18 01:29:56.457343 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:56.457319 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-feded4-5946cb944b-qnvdv"] Jul 18 01:29:56.459185 ip-10-0-134-239 kubenswrapper[2580]: W0718 01:29:56.459150 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb52ae522_c264_467f_85c5_bc0ff444dbf8.slice/crio-4f2cba5222862341d7e84eeace4cfeb78aaec470adda26df7141fee4dbc54b7d WatchSource:0}: Error finding container 4f2cba5222862341d7e84eeace4cfeb78aaec470adda26df7141fee4dbc54b7d: Status 404 returned error can't find the container with id 4f2cba5222862341d7e84eeace4cfeb78aaec470adda26df7141fee4dbc54b7d Jul 18 01:29:56.460590 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:56.460568 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 01:29:56.956687 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:56.956633 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-feded4-5946cb944b-qnvdv" event={"ID":"b52ae522-c264-467f-85c5-bc0ff444dbf8","Type":"ContainerStarted","Data":"4f2cba5222862341d7e84eeace4cfeb78aaec470adda26df7141fee4dbc54b7d"} Jul 18 01:29:57.963969 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:57.963934 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-feded4-5946cb944b-qnvdv" event={"ID":"b52ae522-c264-467f-85c5-bc0ff444dbf8","Type":"ContainerStarted","Data":"0c0069490bb7b57a9c1be2d7578119a4d9784d285f258282d9b0fba25e87eda5"} Jul 18 01:29:57.964379 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:57.964001 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-feded4-5946cb944b-qnvdv" Jul 18 01:29:57.981013 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:29:57.980955 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-feded4-5946cb944b-qnvdv" podStartSLOduration=2.546413932 podStartE2EDuration="2.980939105s" totalImagesPullingTime="434.525173ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:29:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:29:57.979725296 +0000 UTC m=+2958.331575642" watchObservedRunningTime="2026-07-18 01:29:57.980939105 +0000 UTC m=+2958.332789449" Jul 18 01:30:03.400073 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:03.399991 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r"] Jul 18 01:30:03.403959 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:03.403939 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r" Jul 18 01:30:03.406539 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:03.406517 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-8775cd-serving-cert\"" Jul 18 01:30:03.416329 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:03.416290 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r"] Jul 18 01:30:03.514208 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:03.514172 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/150c0317-6215-4860-ae1b-ddbea9a69c6e-maas-api-tls\") pod \"maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r\" (UID: \"150c0317-6215-4860-ae1b-ddbea9a69c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r" Jul 18 01:30:03.514389 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:03.514234 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/150c0317-6215-4860-ae1b-ddbea9a69c6e-openshift-service-ca\") pod \"maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r\" (UID: \"150c0317-6215-4860-ae1b-ddbea9a69c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r" Jul 18 01:30:03.514389 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:03.514287 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx22g\" (UniqueName: \"kubernetes.io/projected/150c0317-6215-4860-ae1b-ddbea9a69c6e-kube-api-access-qx22g\") pod \"maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r\" (UID: \"150c0317-6215-4860-ae1b-ddbea9a69c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r" Jul 18 01:30:03.615154 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:03.615116 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/150c0317-6215-4860-ae1b-ddbea9a69c6e-maas-api-tls\") pod \"maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r\" (UID: \"150c0317-6215-4860-ae1b-ddbea9a69c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r" Jul 18 01:30:03.615328 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:03.615172 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/150c0317-6215-4860-ae1b-ddbea9a69c6e-openshift-service-ca\") pod \"maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r\" (UID: \"150c0317-6215-4860-ae1b-ddbea9a69c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r" Jul 18 01:30:03.615328 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:03.615216 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx22g\" (UniqueName: \"kubernetes.io/projected/150c0317-6215-4860-ae1b-ddbea9a69c6e-kube-api-access-qx22g\") pod \"maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r\" (UID: \"150c0317-6215-4860-ae1b-ddbea9a69c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r" Jul 18 01:30:03.615945 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:03.615926 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/150c0317-6215-4860-ae1b-ddbea9a69c6e-openshift-service-ca\") pod \"maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r\" (UID: \"150c0317-6215-4860-ae1b-ddbea9a69c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r" Jul 18 01:30:03.617729 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:03.617711 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/150c0317-6215-4860-ae1b-ddbea9a69c6e-maas-api-tls\") pod \"maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r\" (UID: \"150c0317-6215-4860-ae1b-ddbea9a69c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r" Jul 18 01:30:03.623434 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:03.623398 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx22g\" (UniqueName: \"kubernetes.io/projected/150c0317-6215-4860-ae1b-ddbea9a69c6e-kube-api-access-qx22g\") pod \"maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r\" (UID: \"150c0317-6215-4860-ae1b-ddbea9a69c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r" Jul 18 01:30:03.717054 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:03.717021 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r" Jul 18 01:30:03.854782 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:03.854758 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r"] Jul 18 01:30:03.856817 ip-10-0-134-239 kubenswrapper[2580]: W0718 01:30:03.856791 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150c0317_6215_4860_ae1b_ddbea9a69c6e.slice/crio-2360592c84af578635c63a3508f22e506e21bc0fe35c162a6fae9298682c9219 WatchSource:0}: Error finding container 2360592c84af578635c63a3508f22e506e21bc0fe35c162a6fae9298682c9219: Status 404 returned error can't find the container with id 2360592c84af578635c63a3508f22e506e21bc0fe35c162a6fae9298682c9219 Jul 18 01:30:03.975145 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:03.975062 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-feded4-5946cb944b-qnvdv" Jul 18 01:30:03.998088 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:03.998057 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r" event={"ID":"150c0317-6215-4860-ae1b-ddbea9a69c6e","Type":"ContainerStarted","Data":"2360592c84af578635c63a3508f22e506e21bc0fe35c162a6fae9298682c9219"} Jul 18 01:30:05.005022 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:05.004987 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r" event={"ID":"150c0317-6215-4860-ae1b-ddbea9a69c6e","Type":"ContainerStarted","Data":"bfe7961295cdf7ea1f6a7c3d0a0aaef37aeeab996df5381e4f75311879c95b84"} Jul 18 01:30:05.005408 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:05.005056 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r" Jul 18 01:30:05.023481 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:05.023430 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r" podStartSLOduration=1.7262912940000001 podStartE2EDuration="2.023414739s" totalImagesPullingTime="297.123445ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:30:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:30:05.022688399 +0000 UTC m=+2965.374538736" watchObservedRunningTime="2026-07-18 01:30:05.023414739 +0000 UTC m=+2965.375265083" Jul 18 01:30:11.016186 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:11.016155 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r" Jul 18 01:30:26.680266 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:26.680224 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-feded4-5946cb944b-qnvdv"] Jul 18 01:30:26.680755 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:26.680469 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-feded4-5946cb944b-qnvdv" podUID="b52ae522-c264-467f-85c5-bc0ff444dbf8" containerName="maas-api" containerID="cri-o://0c0069490bb7b57a9c1be2d7578119a4d9784d285f258282d9b0fba25e87eda5" gracePeriod=30 Jul 18 01:30:26.931069 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:26.931009 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-feded4-5946cb944b-qnvdv" Jul 18 01:30:27.026685 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:27.026643 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b52ae522-c264-467f-85c5-bc0ff444dbf8-openshift-service-ca\") pod \"b52ae522-c264-467f-85c5-bc0ff444dbf8\" (UID: \"b52ae522-c264-467f-85c5-bc0ff444dbf8\") " Jul 18 01:30:27.026922 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:27.026717 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clkg9\" (UniqueName: \"kubernetes.io/projected/b52ae522-c264-467f-85c5-bc0ff444dbf8-kube-api-access-clkg9\") pod \"b52ae522-c264-467f-85c5-bc0ff444dbf8\" (UID: \"b52ae522-c264-467f-85c5-bc0ff444dbf8\") " Jul 18 01:30:27.026922 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:27.026879 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b52ae522-c264-467f-85c5-bc0ff444dbf8-maas-api-tls\") pod \"b52ae522-c264-467f-85c5-bc0ff444dbf8\" (UID: \"b52ae522-c264-467f-85c5-bc0ff444dbf8\") " Jul 18 01:30:27.027096 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:27.027067 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b52ae522-c264-467f-85c5-bc0ff444dbf8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b52ae522-c264-467f-85c5-bc0ff444dbf8" (UID: "b52ae522-c264-467f-85c5-bc0ff444dbf8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 01:30:27.027179 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:27.027163 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b52ae522-c264-467f-85c5-bc0ff444dbf8-openshift-service-ca\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:30:27.028957 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:27.028924 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b52ae522-c264-467f-85c5-bc0ff444dbf8-kube-api-access-clkg9" (OuterVolumeSpecName: "kube-api-access-clkg9") pod "b52ae522-c264-467f-85c5-bc0ff444dbf8" (UID: "b52ae522-c264-467f-85c5-bc0ff444dbf8"). InnerVolumeSpecName "kube-api-access-clkg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 01:30:27.029085 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:27.029032 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b52ae522-c264-467f-85c5-bc0ff444dbf8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b52ae522-c264-467f-85c5-bc0ff444dbf8" (UID: "b52ae522-c264-467f-85c5-bc0ff444dbf8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 01:30:27.119981 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:27.119941 2580 generic.go:356] "Generic (PLEG): container finished" podID="b52ae522-c264-467f-85c5-bc0ff444dbf8" containerID="0c0069490bb7b57a9c1be2d7578119a4d9784d285f258282d9b0fba25e87eda5" exitCode=0 Jul 18 01:30:27.120150 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:27.119988 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-feded4-5946cb944b-qnvdv" Jul 18 01:30:27.120150 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:27.120032 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-feded4-5946cb944b-qnvdv" event={"ID":"b52ae522-c264-467f-85c5-bc0ff444dbf8","Type":"ContainerDied","Data":"0c0069490bb7b57a9c1be2d7578119a4d9784d285f258282d9b0fba25e87eda5"} Jul 18 01:30:27.120150 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:27.120088 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-feded4-5946cb944b-qnvdv" event={"ID":"b52ae522-c264-467f-85c5-bc0ff444dbf8","Type":"ContainerDied","Data":"4f2cba5222862341d7e84eeace4cfeb78aaec470adda26df7141fee4dbc54b7d"} Jul 18 01:30:27.120150 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:27.120109 2580 scope.go:117] "RemoveContainer" containerID="0c0069490bb7b57a9c1be2d7578119a4d9784d285f258282d9b0fba25e87eda5" Jul 18 01:30:27.127936 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:27.127860 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clkg9\" (UniqueName: \"kubernetes.io/projected/b52ae522-c264-467f-85c5-bc0ff444dbf8-kube-api-access-clkg9\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:30:27.127936 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:27.127884 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b52ae522-c264-467f-85c5-bc0ff444dbf8-maas-api-tls\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:30:27.129382 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:27.129366 2580 scope.go:117] "RemoveContainer" containerID="0c0069490bb7b57a9c1be2d7578119a4d9784d285f258282d9b0fba25e87eda5" Jul 18 01:30:27.129597 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:30:27.129578 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c0069490bb7b57a9c1be2d7578119a4d9784d285f258282d9b0fba25e87eda5\": container with ID starting with 0c0069490bb7b57a9c1be2d7578119a4d9784d285f258282d9b0fba25e87eda5 not found: ID does not exist" containerID="0c0069490bb7b57a9c1be2d7578119a4d9784d285f258282d9b0fba25e87eda5" Jul 18 01:30:27.129650 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:27.129610 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c0069490bb7b57a9c1be2d7578119a4d9784d285f258282d9b0fba25e87eda5"} err="failed to get container status \"0c0069490bb7b57a9c1be2d7578119a4d9784d285f258282d9b0fba25e87eda5\": rpc error: code = NotFound desc = could not find container \"0c0069490bb7b57a9c1be2d7578119a4d9784d285f258282d9b0fba25e87eda5\": container with ID starting with 0c0069490bb7b57a9c1be2d7578119a4d9784d285f258282d9b0fba25e87eda5 not found: ID does not exist" Jul 18 01:30:27.141359 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:27.141336 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-feded4-5946cb944b-qnvdv"] Jul 18 01:30:27.146172 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:27.146147 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-feded4-5946cb944b-qnvdv"] Jul 18 01:30:28.148688 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:28.148646 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b52ae522-c264-467f-85c5-bc0ff444dbf8" path="/var/lib/kubelet/pods/b52ae522-c264-467f-85c5-bc0ff444dbf8/volumes" Jul 18 01:30:52.484032 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:52.483991 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r"] Jul 18 01:30:52.484515 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:52.484292 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r" podUID="150c0317-6215-4860-ae1b-ddbea9a69c6e" containerName="maas-api" containerID="cri-o://bfe7961295cdf7ea1f6a7c3d0a0aaef37aeeab996df5381e4f75311879c95b84" gracePeriod=30 Jul 18 01:30:52.737258 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:52.737190 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r" Jul 18 01:30:52.869332 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:52.869296 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/150c0317-6215-4860-ae1b-ddbea9a69c6e-maas-api-tls\") pod \"150c0317-6215-4860-ae1b-ddbea9a69c6e\" (UID: \"150c0317-6215-4860-ae1b-ddbea9a69c6e\") " Jul 18 01:30:52.869514 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:52.869370 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qx22g\" (UniqueName: \"kubernetes.io/projected/150c0317-6215-4860-ae1b-ddbea9a69c6e-kube-api-access-qx22g\") pod \"150c0317-6215-4860-ae1b-ddbea9a69c6e\" (UID: \"150c0317-6215-4860-ae1b-ddbea9a69c6e\") " Jul 18 01:30:52.869514 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:52.869464 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/150c0317-6215-4860-ae1b-ddbea9a69c6e-openshift-service-ca\") pod \"150c0317-6215-4860-ae1b-ddbea9a69c6e\" (UID: \"150c0317-6215-4860-ae1b-ddbea9a69c6e\") " Jul 18 01:30:52.869896 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:52.869873 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/150c0317-6215-4860-ae1b-ddbea9a69c6e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "150c0317-6215-4860-ae1b-ddbea9a69c6e" (UID: "150c0317-6215-4860-ae1b-ddbea9a69c6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 01:30:52.871583 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:52.871559 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/150c0317-6215-4860-ae1b-ddbea9a69c6e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "150c0317-6215-4860-ae1b-ddbea9a69c6e" (UID: "150c0317-6215-4860-ae1b-ddbea9a69c6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 01:30:52.871685 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:52.871614 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/150c0317-6215-4860-ae1b-ddbea9a69c6e-kube-api-access-qx22g" (OuterVolumeSpecName: "kube-api-access-qx22g") pod "150c0317-6215-4860-ae1b-ddbea9a69c6e" (UID: "150c0317-6215-4860-ae1b-ddbea9a69c6e"). InnerVolumeSpecName "kube-api-access-qx22g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 01:30:52.970620 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:52.970570 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx22g\" (UniqueName: \"kubernetes.io/projected/150c0317-6215-4860-ae1b-ddbea9a69c6e-kube-api-access-qx22g\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:30:52.970620 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:52.970608 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/150c0317-6215-4860-ae1b-ddbea9a69c6e-openshift-service-ca\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:30:52.970620 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:52.970618 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/150c0317-6215-4860-ae1b-ddbea9a69c6e-maas-api-tls\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:30:53.255133 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:53.255091 2580 generic.go:356] "Generic (PLEG): container finished" podID="150c0317-6215-4860-ae1b-ddbea9a69c6e" containerID="bfe7961295cdf7ea1f6a7c3d0a0aaef37aeeab996df5381e4f75311879c95b84" exitCode=0 Jul 18 01:30:53.255314 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:53.255139 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r" Jul 18 01:30:53.255314 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:53.255143 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r" event={"ID":"150c0317-6215-4860-ae1b-ddbea9a69c6e","Type":"ContainerDied","Data":"bfe7961295cdf7ea1f6a7c3d0a0aaef37aeeab996df5381e4f75311879c95b84"} Jul 18 01:30:53.255314 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:53.255196 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r" event={"ID":"150c0317-6215-4860-ae1b-ddbea9a69c6e","Type":"ContainerDied","Data":"2360592c84af578635c63a3508f22e506e21bc0fe35c162a6fae9298682c9219"} Jul 18 01:30:53.255314 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:53.255222 2580 scope.go:117] "RemoveContainer" containerID="bfe7961295cdf7ea1f6a7c3d0a0aaef37aeeab996df5381e4f75311879c95b84" Jul 18 01:30:53.266587 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:53.266560 2580 scope.go:117] "RemoveContainer" containerID="bfe7961295cdf7ea1f6a7c3d0a0aaef37aeeab996df5381e4f75311879c95b84" Jul 18 01:30:53.267133 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:30:53.267107 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfe7961295cdf7ea1f6a7c3d0a0aaef37aeeab996df5381e4f75311879c95b84\": container with ID starting with bfe7961295cdf7ea1f6a7c3d0a0aaef37aeeab996df5381e4f75311879c95b84 not found: ID does not exist" containerID="bfe7961295cdf7ea1f6a7c3d0a0aaef37aeeab996df5381e4f75311879c95b84" Jul 18 01:30:53.267232 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:53.267144 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfe7961295cdf7ea1f6a7c3d0a0aaef37aeeab996df5381e4f75311879c95b84"} err="failed to get container status \"bfe7961295cdf7ea1f6a7c3d0a0aaef37aeeab996df5381e4f75311879c95b84\": rpc error: code = NotFound desc = could not find container \"bfe7961295cdf7ea1f6a7c3d0a0aaef37aeeab996df5381e4f75311879c95b84\": container with ID starting with bfe7961295cdf7ea1f6a7c3d0a0aaef37aeeab996df5381e4f75311879c95b84 not found: ID does not exist" Jul 18 01:30:53.281349 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:53.281318 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r"] Jul 18 01:30:53.283482 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:53.283457 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8775cd-5cb4b98fdf-7ms7r"] Jul 18 01:30:54.149791 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:30:54.149757 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="150c0317-6215-4860-ae1b-ddbea9a69c6e" path="/var/lib/kubelet/pods/150c0317-6215-4860-ae1b-ddbea9a69c6e/volumes" Jul 18 01:31:11.728034 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:11.727991 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v"] Jul 18 01:31:11.728554 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:11.728432 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="150c0317-6215-4860-ae1b-ddbea9a69c6e" containerName="maas-api" Jul 18 01:31:11.728554 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:11.728451 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="150c0317-6215-4860-ae1b-ddbea9a69c6e" containerName="maas-api" Jul 18 01:31:11.728554 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:11.728468 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b52ae522-c264-467f-85c5-bc0ff444dbf8" containerName="maas-api" Jul 18 01:31:11.728554 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:11.728476 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b52ae522-c264-467f-85c5-bc0ff444dbf8" containerName="maas-api" Jul 18 01:31:11.728851 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:11.728561 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b52ae522-c264-467f-85c5-bc0ff444dbf8" containerName="maas-api" Jul 18 01:31:11.728851 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:11.728577 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="150c0317-6215-4860-ae1b-ddbea9a69c6e" containerName="maas-api" Jul 18 01:31:11.733154 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:11.733129 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v" Jul 18 01:31:11.733577 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:11.733539 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18964af8-1d76-4c70-96eb-4e977ec59872-maas-api-tls\") pod \"maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v\" (UID: \"18964af8-1d76-4c70-96eb-4e977ec59872\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v" Jul 18 01:31:11.733721 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:11.733613 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18964af8-1d76-4c70-96eb-4e977ec59872-openshift-service-ca\") pod \"maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v\" (UID: \"18964af8-1d76-4c70-96eb-4e977ec59872\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v" Jul 18 01:31:11.733721 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:11.733684 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nndp9\" (UniqueName: \"kubernetes.io/projected/18964af8-1d76-4c70-96eb-4e977ec59872-kube-api-access-nndp9\") pod \"maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v\" (UID: \"18964af8-1d76-4c70-96eb-4e977ec59872\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v" Jul 18 01:31:11.735915 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:11.735888 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-6b3233-serving-cert\"" Jul 18 01:31:11.736066 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:11.735966 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2djvc\"" Jul 18 01:31:11.746346 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:11.746315 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v"] Jul 18 01:31:11.834768 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:11.834723 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18964af8-1d76-4c70-96eb-4e977ec59872-openshift-service-ca\") pod \"maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v\" (UID: \"18964af8-1d76-4c70-96eb-4e977ec59872\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v" Jul 18 01:31:11.834985 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:11.834785 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nndp9\" (UniqueName: \"kubernetes.io/projected/18964af8-1d76-4c70-96eb-4e977ec59872-kube-api-access-nndp9\") pod \"maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v\" (UID: \"18964af8-1d76-4c70-96eb-4e977ec59872\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v" Jul 18 01:31:11.834985 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:11.834870 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18964af8-1d76-4c70-96eb-4e977ec59872-maas-api-tls\") pod \"maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v\" (UID: \"18964af8-1d76-4c70-96eb-4e977ec59872\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v" Jul 18 01:31:11.835553 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:11.835526 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18964af8-1d76-4c70-96eb-4e977ec59872-openshift-service-ca\") pod \"maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v\" (UID: \"18964af8-1d76-4c70-96eb-4e977ec59872\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v" Jul 18 01:31:11.837438 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:11.837410 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18964af8-1d76-4c70-96eb-4e977ec59872-maas-api-tls\") pod \"maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v\" (UID: \"18964af8-1d76-4c70-96eb-4e977ec59872\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v" Jul 18 01:31:11.843959 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:11.843936 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nndp9\" (UniqueName: \"kubernetes.io/projected/18964af8-1d76-4c70-96eb-4e977ec59872-kube-api-access-nndp9\") pod \"maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v\" (UID: \"18964af8-1d76-4c70-96eb-4e977ec59872\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v" Jul 18 01:31:12.046419 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:12.046329 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v" Jul 18 01:31:12.193163 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:12.193135 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v"] Jul 18 01:31:12.195089 ip-10-0-134-239 kubenswrapper[2580]: W0718 01:31:12.195063 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18964af8_1d76_4c70_96eb_4e977ec59872.slice/crio-6faa85f2ece813b916b23b75096bd094e782e29a2cb084e0e45ed25d4846e4aa WatchSource:0}: Error finding container 6faa85f2ece813b916b23b75096bd094e782e29a2cb084e0e45ed25d4846e4aa: Status 404 returned error can't find the container with id 6faa85f2ece813b916b23b75096bd094e782e29a2cb084e0e45ed25d4846e4aa Jul 18 01:31:12.356840 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:12.356731 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v" event={"ID":"18964af8-1d76-4c70-96eb-4e977ec59872","Type":"ContainerStarted","Data":"6faa85f2ece813b916b23b75096bd094e782e29a2cb084e0e45ed25d4846e4aa"} Jul 18 01:31:13.364821 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:13.364787 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v" event={"ID":"18964af8-1d76-4c70-96eb-4e977ec59872","Type":"ContainerStarted","Data":"94bb93fa45b4cfbdf7ac4dd52b2e68c7702603be8c95e83c54b8f68464fae788"} Jul 18 01:31:13.365287 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:13.364918 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v" Jul 18 01:31:13.386638 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:13.386557 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v" podStartSLOduration=2.045185513 podStartE2EDuration="2.386539237s" totalImagesPullingTime="341.353724ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:31:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:31:13.385185061 +0000 UTC m=+3033.737035409" watchObservedRunningTime="2026-07-18 01:31:13.386539237 +0000 UTC m=+3033.738389583" Jul 18 01:31:19.375361 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:19.375323 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v" Jul 18 01:31:19.805140 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:19.805100 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz"] Jul 18 01:31:19.809044 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:19.809018 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz" Jul 18 01:31:19.811291 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:19.811264 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-b17102-serving-cert\"" Jul 18 01:31:19.818512 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:19.818330 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz"] Jul 18 01:31:19.895435 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:19.895394 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc7zl\" (UniqueName: \"kubernetes.io/projected/1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0-kube-api-access-tc7zl\") pod \"maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz\" (UID: \"1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz" Jul 18 01:31:19.895626 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:19.895457 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz\" (UID: \"1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz" Jul 18 01:31:19.895626 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:19.895561 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0-maas-api-tls\") pod \"maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz\" (UID: \"1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz" Jul 18 01:31:19.996593 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:19.996544 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc7zl\" (UniqueName: \"kubernetes.io/projected/1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0-kube-api-access-tc7zl\") pod \"maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz\" (UID: \"1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz" Jul 18 01:31:19.996823 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:19.996612 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz\" (UID: \"1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz" Jul 18 01:31:19.996823 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:19.996682 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0-maas-api-tls\") pod \"maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz\" (UID: \"1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz" Jul 18 01:31:19.996823 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:31:19.996804 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-b17102-serving-cert: secret "maas-api-e2e-shared-b-b17102-serving-cert" not found Jul 18 01:31:19.997001 ip-10-0-134-239 kubenswrapper[2580]: E0718 01:31:19.996876 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0-maas-api-tls podName:1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0 nodeName:}" failed. No retries permitted until 2026-07-18 01:31:20.496853969 +0000 UTC m=+3040.848704315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0-maas-api-tls") pod "maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz" (UID: "1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0") : secret "maas-api-e2e-shared-b-b17102-serving-cert" not found Jul 18 01:31:19.997377 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:19.997353 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz\" (UID: \"1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz" Jul 18 01:31:20.006782 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:20.006752 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc7zl\" (UniqueName: \"kubernetes.io/projected/1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0-kube-api-access-tc7zl\") pod \"maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz\" (UID: \"1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz" Jul 18 01:31:20.501569 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:20.501524 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0-maas-api-tls\") pod \"maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz\" (UID: \"1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz" Jul 18 01:31:20.504135 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:20.504112 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0-maas-api-tls\") pod \"maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz\" (UID: \"1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz" Jul 18 01:31:20.723146 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:20.723101 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz" Jul 18 01:31:20.885114 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:20.885088 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz"] Jul 18 01:31:20.886794 ip-10-0-134-239 kubenswrapper[2580]: W0718 01:31:20.886767 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f7e5e70_58c1_4a0f_9c52_f3c9194c44f0.slice/crio-29bb65672b5235579f1273cec4c825fb32dd41d4e87817a0b2ab8384bd557128 WatchSource:0}: Error finding container 29bb65672b5235579f1273cec4c825fb32dd41d4e87817a0b2ab8384bd557128: Status 404 returned error can't find the container with id 29bb65672b5235579f1273cec4c825fb32dd41d4e87817a0b2ab8384bd557128 Jul 18 01:31:21.414740 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:21.414700 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz" event={"ID":"1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0","Type":"ContainerStarted","Data":"e3ef1730754f2c37fff64f861ae533c5c7135fa4b479a9816bae2e8ac92991cd"} Jul 18 01:31:21.414933 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:21.414751 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz" event={"ID":"1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0","Type":"ContainerStarted","Data":"29bb65672b5235579f1273cec4c825fb32dd41d4e87817a0b2ab8384bd557128"} Jul 18 01:31:21.414933 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:21.414900 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz" Jul 18 01:31:21.430833 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:21.430775 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz" podStartSLOduration=2.125682705 podStartE2EDuration="2.430758468s" totalImagesPullingTime="305.075763ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 01:31:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:31:21.428865576 +0000 UTC m=+3041.780715921" watchObservedRunningTime="2026-07-18 01:31:21.430758468 +0000 UTC m=+3041.782608812" Jul 18 01:31:27.425787 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:31:27.425752 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz" Jul 18 01:37:54.247356 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:37:54.247269 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v"] Jul 18 01:37:54.248159 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:37:54.247535 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v" podUID="18964af8-1d76-4c70-96eb-4e977ec59872" containerName="maas-api" containerID="cri-o://94bb93fa45b4cfbdf7ac4dd52b2e68c7702603be8c95e83c54b8f68464fae788" gracePeriod=30 Jul 18 01:37:54.370989 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:37:54.370952 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v" podUID="18964af8-1d76-4c70-96eb-4e977ec59872" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.54:8443/health\": dial tcp 10.134.0.54:8443: connect: connection refused" Jul 18 01:37:54.431239 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:37:54.431204 2580 generic.go:356] "Generic (PLEG): container finished" podID="18964af8-1d76-4c70-96eb-4e977ec59872" containerID="94bb93fa45b4cfbdf7ac4dd52b2e68c7702603be8c95e83c54b8f68464fae788" exitCode=0 Jul 18 01:37:54.431386 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:37:54.431272 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v" event={"ID":"18964af8-1d76-4c70-96eb-4e977ec59872","Type":"ContainerDied","Data":"94bb93fa45b4cfbdf7ac4dd52b2e68c7702603be8c95e83c54b8f68464fae788"} Jul 18 01:37:54.513379 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:37:54.513355 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v" Jul 18 01:37:54.593937 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:37:54.593901 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nndp9\" (UniqueName: \"kubernetes.io/projected/18964af8-1d76-4c70-96eb-4e977ec59872-kube-api-access-nndp9\") pod \"18964af8-1d76-4c70-96eb-4e977ec59872\" (UID: \"18964af8-1d76-4c70-96eb-4e977ec59872\") " Jul 18 01:37:54.593937 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:37:54.593943 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18964af8-1d76-4c70-96eb-4e977ec59872-openshift-service-ca\") pod \"18964af8-1d76-4c70-96eb-4e977ec59872\" (UID: \"18964af8-1d76-4c70-96eb-4e977ec59872\") " Jul 18 01:37:54.594193 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:37:54.593970 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18964af8-1d76-4c70-96eb-4e977ec59872-maas-api-tls\") pod \"18964af8-1d76-4c70-96eb-4e977ec59872\" (UID: \"18964af8-1d76-4c70-96eb-4e977ec59872\") " Jul 18 01:37:54.594328 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:37:54.594297 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18964af8-1d76-4c70-96eb-4e977ec59872-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "18964af8-1d76-4c70-96eb-4e977ec59872" (UID: "18964af8-1d76-4c70-96eb-4e977ec59872"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 01:37:54.596255 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:37:54.596225 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18964af8-1d76-4c70-96eb-4e977ec59872-kube-api-access-nndp9" (OuterVolumeSpecName: "kube-api-access-nndp9") pod "18964af8-1d76-4c70-96eb-4e977ec59872" (UID: "18964af8-1d76-4c70-96eb-4e977ec59872"). InnerVolumeSpecName "kube-api-access-nndp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 01:37:54.596374 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:37:54.596319 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18964af8-1d76-4c70-96eb-4e977ec59872-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "18964af8-1d76-4c70-96eb-4e977ec59872" (UID: "18964af8-1d76-4c70-96eb-4e977ec59872"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 01:37:54.695052 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:37:54.695018 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nndp9\" (UniqueName: \"kubernetes.io/projected/18964af8-1d76-4c70-96eb-4e977ec59872-kube-api-access-nndp9\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:37:54.695052 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:37:54.695048 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18964af8-1d76-4c70-96eb-4e977ec59872-openshift-service-ca\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:37:54.695247 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:37:54.695063 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18964af8-1d76-4c70-96eb-4e977ec59872-maas-api-tls\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:37:55.437655 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:37:55.437621 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v" event={"ID":"18964af8-1d76-4c70-96eb-4e977ec59872","Type":"ContainerDied","Data":"6faa85f2ece813b916b23b75096bd094e782e29a2cb084e0e45ed25d4846e4aa"} Jul 18 01:37:55.438092 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:37:55.437704 2580 scope.go:117] "RemoveContainer" containerID="94bb93fa45b4cfbdf7ac4dd52b2e68c7702603be8c95e83c54b8f68464fae788" Jul 18 01:37:55.438092 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:37:55.437630 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v" Jul 18 01:37:55.460122 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:37:55.460093 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v"] Jul 18 01:37:55.462178 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:37:55.462151 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6b3233-7f4f6f8dfd-6zw7v"] Jul 18 01:37:56.149098 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:37:56.149061 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18964af8-1d76-4c70-96eb-4e977ec59872" path="/var/lib/kubelet/pods/18964af8-1d76-4c70-96eb-4e977ec59872/volumes" Jul 18 01:38:20.396170 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:20.396129 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz"] Jul 18 01:38:20.396809 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:20.396409 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz" podUID="1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0" containerName="maas-api" containerID="cri-o://e3ef1730754f2c37fff64f861ae533c5c7135fa4b479a9816bae2e8ac92991cd" gracePeriod=30 Jul 18 01:38:20.564114 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:20.564082 2580 generic.go:356] "Generic (PLEG): container finished" podID="1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0" containerID="e3ef1730754f2c37fff64f861ae533c5c7135fa4b479a9816bae2e8ac92991cd" exitCode=0 Jul 18 01:38:20.564285 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:20.564161 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz" event={"ID":"1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0","Type":"ContainerDied","Data":"e3ef1730754f2c37fff64f861ae533c5c7135fa4b479a9816bae2e8ac92991cd"} Jul 18 01:38:20.650364 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:20.650299 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz" Jul 18 01:38:20.737119 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:20.737078 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tc7zl\" (UniqueName: \"kubernetes.io/projected/1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0-kube-api-access-tc7zl\") pod \"1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0\" (UID: \"1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0\") " Jul 18 01:38:20.737119 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:20.737128 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0-openshift-service-ca\") pod \"1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0\" (UID: \"1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0\") " Jul 18 01:38:20.737352 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:20.737190 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0-maas-api-tls\") pod \"1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0\" (UID: \"1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0\") " Jul 18 01:38:20.737499 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:20.737476 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0" (UID: "1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 01:38:20.739335 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:20.739313 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0" (UID: "1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 01:38:20.739463 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:20.739444 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0-kube-api-access-tc7zl" (OuterVolumeSpecName: "kube-api-access-tc7zl") pod "1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0" (UID: "1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0"). InnerVolumeSpecName "kube-api-access-tc7zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 01:38:20.838596 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:20.838558 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc7zl\" (UniqueName: \"kubernetes.io/projected/1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0-kube-api-access-tc7zl\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:38:20.838596 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:20.838589 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0-openshift-service-ca\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:38:20.838596 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:20.838598 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0-maas-api-tls\") on node \"ip-10-0-134-239.ec2.internal\" DevicePath \"\"" Jul 18 01:38:21.570748 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:21.570710 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz" Jul 18 01:38:21.570748 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:21.570730 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz" event={"ID":"1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0","Type":"ContainerDied","Data":"29bb65672b5235579f1273cec4c825fb32dd41d4e87817a0b2ab8384bd557128"} Jul 18 01:38:21.571231 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:21.570793 2580 scope.go:117] "RemoveContainer" containerID="e3ef1730754f2c37fff64f861ae533c5c7135fa4b479a9816bae2e8ac92991cd" Jul 18 01:38:21.593143 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:21.593106 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz"] Jul 18 01:38:21.596483 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:21.596457 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b17102-67b7b46fc-dgrzz"] Jul 18 01:38:22.154533 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:22.154498 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0" path="/var/lib/kubelet/pods/1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0/volumes" Jul 18 01:38:42.045351 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:42.045314 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b7ff4fd55-2ltkx_04dd6278-f1d4-4b48-8026-de1958a1123b/manager/0.log" Jul 18 01:38:42.390429 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:42.390355 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w_f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92/copy-manifests/0.log" Jul 18 01:38:42.396727 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:42.396699 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w_f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92/manager/0.log" Jul 18 01:38:43.002366 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:43.002336 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-775b7f98ff-mgk2l_6b54b1f1-dacb-4bf2-baba-ae004b53990f/maas-api/0.log" Jul 18 01:38:44.535728 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:44.535693 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-zbzd6_1d30b4d1-766d-4600-ab4b-3e6851cae623/postgres/0.log" Jul 18 01:38:45.345550 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:45.345521 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb_b775b754-04e1-4a9e-86a8-dc0740adae1a/util/0.log" Jul 18 01:38:45.354643 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:45.354617 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb_b775b754-04e1-4a9e-86a8-dc0740adae1a/pull/0.log" Jul 18 01:38:45.360269 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:45.360253 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb_b775b754-04e1-4a9e-86a8-dc0740adae1a/extract/0.log" Jul 18 01:38:45.476474 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:45.476444 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6_388cea1d-4275-49d1-afd1-d07f2e4bedd2/util/0.log" Jul 18 01:38:45.482222 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:45.482200 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6_388cea1d-4275-49d1-afd1-d07f2e4bedd2/pull/0.log" Jul 18 01:38:45.488044 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:45.488023 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6_388cea1d-4275-49d1-afd1-d07f2e4bedd2/extract/0.log" Jul 18 01:38:45.601341 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:45.601255 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv_69a781e2-a1bd-4732-8007-717d6a3b21b6/util/0.log" Jul 18 01:38:45.606983 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:45.606955 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv_69a781e2-a1bd-4732-8007-717d6a3b21b6/pull/0.log" Jul 18 01:38:45.612371 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:45.612350 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv_69a781e2-a1bd-4732-8007-717d6a3b21b6/extract/0.log" Jul 18 01:38:45.729486 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:45.729459 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89_928dd7f6-6237-4152-adc0-0126cba4d769/util/0.log" Jul 18 01:38:45.735306 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:45.735278 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89_928dd7f6-6237-4152-adc0-0126cba4d769/pull/0.log" Jul 18 01:38:45.740769 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:45.740745 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89_928dd7f6-6237-4152-adc0-0126cba4d769/extract/0.log" Jul 18 01:38:47.148329 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:47.148295 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-jg4w7_8efec2bf-6c93-4171-9363-772e969aa333/discovery/0.log" Jul 18 01:38:47.717803 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:47.717770 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-jcklw_734a2b3a-9cfa-491c-a66f-c9913a167e6c/payload-processing/0.log" Jul 18 01:38:48.167740 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:48.167642 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-nnl57_7a1845e9-c79e-47a8-a4fa-779f299ff699/main/0.log" Jul 18 01:38:48.399646 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:48.399616 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-tgpsg_a4aba52a-c110-4efd-8b66-5a806a4365c8/main/0.log" Jul 18 01:38:48.623024 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:48.622992 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-w45pv_05b8e38e-1881-4eb7-b299-030868877fc9/main/0.log" Jul 18 01:38:57.230557 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:57.230523 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xltqc_3cb92a57-b715-4664-8342-fe08ec474cc1/global-pull-secret-syncer/0.log" Jul 18 01:38:57.383644 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:57.383608 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mktkv_08f02b87-a5b3-44ef-81bb-37637a148986/konnectivity-agent/0.log" Jul 18 01:38:57.426959 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:38:57.426927 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-239.ec2.internal_481ee2577fe0b9fe2f48bf6abc750a85/haproxy/0.log" Jul 18 01:39:01.893129 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:01.893032 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb_b775b754-04e1-4a9e-86a8-dc0740adae1a/extract/0.log" Jul 18 01:39:01.918218 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:01.918186 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb_b775b754-04e1-4a9e-86a8-dc0740adae1a/util/0.log" Jul 18 01:39:01.940031 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:01.940003 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599cdsb_b775b754-04e1-4a9e-86a8-dc0740adae1a/pull/0.log" Jul 18 01:39:01.968149 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:01.968124 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6_388cea1d-4275-49d1-afd1-d07f2e4bedd2/extract/0.log" Jul 18 01:39:01.994405 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:01.994376 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6_388cea1d-4275-49d1-afd1-d07f2e4bedd2/util/0.log" Jul 18 01:39:02.014718 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:02.014700 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lp4b6_388cea1d-4275-49d1-afd1-d07f2e4bedd2/pull/0.log" Jul 18 01:39:02.039925 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:02.039890 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv_69a781e2-a1bd-4732-8007-717d6a3b21b6/extract/0.log" Jul 18 01:39:02.058882 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:02.058861 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv_69a781e2-a1bd-4732-8007-717d6a3b21b6/util/0.log" Jul 18 01:39:02.085795 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:02.085768 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73rvsqv_69a781e2-a1bd-4732-8007-717d6a3b21b6/pull/0.log" Jul 18 01:39:02.123219 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:02.123192 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89_928dd7f6-6237-4152-adc0-0126cba4d769/extract/0.log" Jul 18 01:39:02.148465 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:02.148395 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89_928dd7f6-6237-4152-adc0-0126cba4d769/util/0.log" Jul 18 01:39:02.166686 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:02.166644 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bbf89_928dd7f6-6237-4152-adc0-0126cba4d769/pull/0.log" Jul 18 01:39:05.253215 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:05.253184 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8nrwv_2892ae85-8898-4539-b4ee-0be960d2da88/node-exporter/0.log" Jul 18 01:39:05.274511 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:05.274487 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8nrwv_2892ae85-8898-4539-b4ee-0be960d2da88/kube-rbac-proxy/0.log" Jul 18 01:39:05.296747 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:05.296718 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8nrwv_2892ae85-8898-4539-b4ee-0be960d2da88/init-textfile/0.log" Jul 18 01:39:05.648611 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:05.648530 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-xl64x_4eefbcad-4399-4d17-9d1a-acd7348c2380/prometheus-operator/0.log" Jul 18 01:39:05.667064 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:05.667037 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-xl64x_4eefbcad-4399-4d17-9d1a-acd7348c2380/kube-rbac-proxy/0.log" Jul 18 01:39:05.694171 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:05.694139 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-v2nl7_522140e6-dd37-4a6f-8c78-e95949b15f4b/prometheus-operator-admission-webhook/0.log" Jul 18 01:39:05.723057 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:05.723031 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-65989b4df6-95szt_a6f19b7d-7b8c-4e65-ae49-a284bd1191e1/telemeter-client/0.log" Jul 18 01:39:05.746765 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:05.746727 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-65989b4df6-95szt_a6f19b7d-7b8c-4e65-ae49-a284bd1191e1/reload/0.log" Jul 18 01:39:05.765990 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:05.765966 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-65989b4df6-95szt_a6f19b7d-7b8c-4e65-ae49-a284bd1191e1/kube-rbac-proxy/0.log" Jul 18 01:39:07.507114 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:07.507083 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dn4z4/perf-node-gather-daemonset-kp79q"] Jul 18 01:39:07.507609 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:07.507589 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18964af8-1d76-4c70-96eb-4e977ec59872" containerName="maas-api" Jul 18 01:39:07.507749 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:07.507613 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="18964af8-1d76-4c70-96eb-4e977ec59872" containerName="maas-api" Jul 18 01:39:07.507749 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:07.507625 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0" containerName="maas-api" Jul 18 01:39:07.507749 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:07.507633 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0" containerName="maas-api" Jul 18 01:39:07.507906 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:07.507765 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="18964af8-1d76-4c70-96eb-4e977ec59872" containerName="maas-api" Jul 18 01:39:07.507906 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:07.507784 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f7e5e70-58c1-4a0f-9c52-f3c9194c44f0" containerName="maas-api" Jul 18 01:39:07.510933 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:07.510911 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dn4z4/perf-node-gather-daemonset-kp79q" Jul 18 01:39:07.513319 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:07.513297 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dn4z4\"/\"default-dockercfg-glxnk\"" Jul 18 01:39:07.514088 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:07.514072 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dn4z4\"/\"openshift-service-ca.crt\"" Jul 18 01:39:07.514186 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:07.514086 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dn4z4\"/\"kube-root-ca.crt\"" Jul 18 01:39:07.520730 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:07.520705 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dn4z4/perf-node-gather-daemonset-kp79q"] Jul 18 01:39:07.550040 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:07.549993 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/080ed171-23bd-4ea0-a837-17c92b6ef392-proc\") pod \"perf-node-gather-daemonset-kp79q\" (UID: \"080ed171-23bd-4ea0-a837-17c92b6ef392\") " pod="openshift-must-gather-dn4z4/perf-node-gather-daemonset-kp79q" Jul 18 01:39:07.550245 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:07.550066 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/080ed171-23bd-4ea0-a837-17c92b6ef392-podres\") pod \"perf-node-gather-daemonset-kp79q\" (UID: \"080ed171-23bd-4ea0-a837-17c92b6ef392\") " pod="openshift-must-gather-dn4z4/perf-node-gather-daemonset-kp79q" Jul 18 01:39:07.550245 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:07.550106 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/080ed171-23bd-4ea0-a837-17c92b6ef392-sys\") pod \"perf-node-gather-daemonset-kp79q\" (UID: \"080ed171-23bd-4ea0-a837-17c92b6ef392\") " pod="openshift-must-gather-dn4z4/perf-node-gather-daemonset-kp79q" Jul 18 01:39:07.550245 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:07.550167 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/080ed171-23bd-4ea0-a837-17c92b6ef392-lib-modules\") pod \"perf-node-gather-daemonset-kp79q\" (UID: \"080ed171-23bd-4ea0-a837-17c92b6ef392\") " pod="openshift-must-gather-dn4z4/perf-node-gather-daemonset-kp79q" Jul 18 01:39:07.550414 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:07.550256 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq5bp\" (UniqueName: \"kubernetes.io/projected/080ed171-23bd-4ea0-a837-17c92b6ef392-kube-api-access-dq5bp\") pod \"perf-node-gather-daemonset-kp79q\" (UID: \"080ed171-23bd-4ea0-a837-17c92b6ef392\") " pod="openshift-must-gather-dn4z4/perf-node-gather-daemonset-kp79q" Jul 18 01:39:07.651464 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:07.651423 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/080ed171-23bd-4ea0-a837-17c92b6ef392-podres\") pod \"perf-node-gather-daemonset-kp79q\" (UID: \"080ed171-23bd-4ea0-a837-17c92b6ef392\") " pod="openshift-must-gather-dn4z4/perf-node-gather-daemonset-kp79q" Jul 18 01:39:07.651464 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:07.651462 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/080ed171-23bd-4ea0-a837-17c92b6ef392-sys\") pod \"perf-node-gather-daemonset-kp79q\" (UID: \"080ed171-23bd-4ea0-a837-17c92b6ef392\") " pod="openshift-must-gather-dn4z4/perf-node-gather-daemonset-kp79q" Jul 18 01:39:07.651713 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:07.651506 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/080ed171-23bd-4ea0-a837-17c92b6ef392-lib-modules\") pod \"perf-node-gather-daemonset-kp79q\" (UID: \"080ed171-23bd-4ea0-a837-17c92b6ef392\") " pod="openshift-must-gather-dn4z4/perf-node-gather-daemonset-kp79q" Jul 18 01:39:07.651713 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:07.651569 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq5bp\" (UniqueName: \"kubernetes.io/projected/080ed171-23bd-4ea0-a837-17c92b6ef392-kube-api-access-dq5bp\") pod \"perf-node-gather-daemonset-kp79q\" (UID: \"080ed171-23bd-4ea0-a837-17c92b6ef392\") " pod="openshift-must-gather-dn4z4/perf-node-gather-daemonset-kp79q" Jul 18 01:39:07.651713 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:07.651616 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/080ed171-23bd-4ea0-a837-17c92b6ef392-podres\") pod \"perf-node-gather-daemonset-kp79q\" (UID: \"080ed171-23bd-4ea0-a837-17c92b6ef392\") " pod="openshift-must-gather-dn4z4/perf-node-gather-daemonset-kp79q" Jul 18 01:39:07.651713 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:07.651627 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/080ed171-23bd-4ea0-a837-17c92b6ef392-sys\") pod \"perf-node-gather-daemonset-kp79q\" (UID: \"080ed171-23bd-4ea0-a837-17c92b6ef392\") " pod="openshift-must-gather-dn4z4/perf-node-gather-daemonset-kp79q" Jul 18 01:39:07.651713 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:07.651621 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/080ed171-23bd-4ea0-a837-17c92b6ef392-proc\") pod \"perf-node-gather-daemonset-kp79q\" (UID: \"080ed171-23bd-4ea0-a837-17c92b6ef392\") " pod="openshift-must-gather-dn4z4/perf-node-gather-daemonset-kp79q" Jul 18 01:39:07.651713 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:07.651659 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/080ed171-23bd-4ea0-a837-17c92b6ef392-proc\") pod \"perf-node-gather-daemonset-kp79q\" (UID: \"080ed171-23bd-4ea0-a837-17c92b6ef392\") " pod="openshift-must-gather-dn4z4/perf-node-gather-daemonset-kp79q" Jul 18 01:39:07.651713 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:07.651708 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/080ed171-23bd-4ea0-a837-17c92b6ef392-lib-modules\") pod \"perf-node-gather-daemonset-kp79q\" (UID: \"080ed171-23bd-4ea0-a837-17c92b6ef392\") " pod="openshift-must-gather-dn4z4/perf-node-gather-daemonset-kp79q" Jul 18 01:39:07.659836 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:07.659811 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq5bp\" (UniqueName: \"kubernetes.io/projected/080ed171-23bd-4ea0-a837-17c92b6ef392-kube-api-access-dq5bp\") pod \"perf-node-gather-daemonset-kp79q\" (UID: \"080ed171-23bd-4ea0-a837-17c92b6ef392\") " pod="openshift-must-gather-dn4z4/perf-node-gather-daemonset-kp79q" Jul 18 01:39:07.822557 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:07.822467 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dn4z4/perf-node-gather-daemonset-kp79q" Jul 18 01:39:08.026223 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:08.026190 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dn4z4/perf-node-gather-daemonset-kp79q"] Jul 18 01:39:08.026585 ip-10-0-134-239 kubenswrapper[2580]: W0718 01:39:08.026562 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod080ed171_23bd_4ea0_a837_17c92b6ef392.slice/crio-6929369c627c7f3e0f76bd640897a553dac88ba80620c9b3878cb4789189b2c5 WatchSource:0}: Error finding container 6929369c627c7f3e0f76bd640897a553dac88ba80620c9b3878cb4789189b2c5: Status 404 returned error can't find the container with id 6929369c627c7f3e0f76bd640897a553dac88ba80620c9b3878cb4789189b2c5 Jul 18 01:39:08.285214 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:08.285184 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5b7b88f5d8-ttdwm_17efbe25-7468-46c3-9df0-cbdf2611bbef/console/0.log" Jul 18 01:39:08.817535 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:08.817497 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dn4z4/perf-node-gather-daemonset-kp79q" event={"ID":"080ed171-23bd-4ea0-a837-17c92b6ef392","Type":"ContainerStarted","Data":"f9ef96738ffc1158ec1953d2f2ca2d3663dda83c0799aea98d54ebc691559603"} Jul 18 01:39:08.818064 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:08.817554 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dn4z4/perf-node-gather-daemonset-kp79q" event={"ID":"080ed171-23bd-4ea0-a837-17c92b6ef392","Type":"ContainerStarted","Data":"6929369c627c7f3e0f76bd640897a553dac88ba80620c9b3878cb4789189b2c5"} Jul 18 01:39:08.818064 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:08.817614 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dn4z4/perf-node-gather-daemonset-kp79q" Jul 18 01:39:08.832829 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:08.832782 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dn4z4/perf-node-gather-daemonset-kp79q" podStartSLOduration=1.832767228 podStartE2EDuration="1.832767228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 01:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 01:39:08.832230882 +0000 UTC m=+3509.184081228" watchObservedRunningTime="2026-07-18 01:39:08.832767228 +0000 UTC m=+3509.184617604" Jul 18 01:39:09.956808 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:09.956780 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dvbgz_e7471583-1515-4d25-afeb-539e6484f43d/dns/0.log" Jul 18 01:39:09.976120 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:09.976093 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dvbgz_e7471583-1515-4d25-afeb-539e6484f43d/kube-rbac-proxy/0.log" Jul 18 01:39:10.093945 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:10.093917 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-59t57_2159fdd6-af24-4d1b-ac04-2f8e41735fc3/dns-node-resolver/0.log" Jul 18 01:39:10.712243 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:10.712212 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7ddb64499c-fkprz_a2502e12-3e9c-4a20-a568-719afe10358e/registry/0.log" Jul 18 01:39:10.776616 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:10.776588 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lpl5x_ed6ea7dc-3703-4727-8b9d-b8671f2e89fb/node-ca/0.log" Jul 18 01:39:11.859833 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:11.859794 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-jg4w7_8efec2bf-6c93-4171-9363-772e969aa333/discovery/0.log" Jul 18 01:39:12.031663 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:12.031635 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-jcklw_734a2b3a-9cfa-491c-a66f-c9913a167e6c/payload-processing/0.log" Jul 18 01:39:12.785491 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:12.785455 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qpq66_1a02dce2-9348-462f-be64-820d42bdd849/serve-healthcheck-canary/0.log" Jul 18 01:39:13.395378 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:13.395347 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-swzld_1fc2c61e-6d9a-4c11-b942-770a3d15153c/kube-rbac-proxy/0.log" Jul 18 01:39:13.414660 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:13.414629 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-swzld_1fc2c61e-6d9a-4c11-b942-770a3d15153c/exporter/0.log" Jul 18 01:39:13.437875 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:13.437847 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-swzld_1fc2c61e-6d9a-4c11-b942-770a3d15153c/extractor/0.log" Jul 18 01:39:14.833188 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:14.833158 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dn4z4/perf-node-gather-daemonset-kp79q" Jul 18 01:39:15.752384 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:15.752352 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b7ff4fd55-2ltkx_04dd6278-f1d4-4b48-8026-de1958a1123b/manager/0.log" Jul 18 01:39:15.882374 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:15.882343 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w_f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92/manager/0.log" Jul 18 01:39:15.903501 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:15.903475 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bd8c7c4c4-29p6w_f3efa0cb-1b7c-44ca-a31b-fd6de43eaa92/copy-manifests/0.log" Jul 18 01:39:17.465169 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:17.465136 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-845f554b8f-989s7_63cbaf43-e409-4e8f-8280-aed86a0594cc/manager/0.log" Jul 18 01:39:17.519375 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:17.519340 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-xx8q4_777066f8-7c6b-4be1-b328-224d160dcd29/openshift-lws-operator/0.log" Jul 18 01:39:18.607998 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:18.607966 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-jg4w7_8efec2bf-6c93-4171-9363-772e969aa333/discovery/0.log" Jul 18 01:39:18.744699 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:18.744655 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-jcklw_734a2b3a-9cfa-491c-a66f-c9913a167e6c/payload-processing/0.log" Jul 18 01:39:20.090045 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:20.090008 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-jg4w7_8efec2bf-6c93-4171-9363-772e969aa333/discovery/0.log" Jul 18 01:39:20.224973 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:20.224939 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-jcklw_734a2b3a-9cfa-491c-a66f-c9913a167e6c/payload-processing/0.log" Jul 18 01:39:24.129988 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:24.129959 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qkpwq_6616a85b-1ee0-492b-8edc-a50d7d416d9d/kube-multus-additional-cni-plugins/0.log" Jul 18 01:39:24.149443 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:24.149415 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qkpwq_6616a85b-1ee0-492b-8edc-a50d7d416d9d/egress-router-binary-copy/0.log" Jul 18 01:39:24.169197 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:24.169166 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qkpwq_6616a85b-1ee0-492b-8edc-a50d7d416d9d/cni-plugins/0.log" Jul 18 01:39:24.188234 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:24.188206 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qkpwq_6616a85b-1ee0-492b-8edc-a50d7d416d9d/bond-cni-plugin/0.log" Jul 18 01:39:24.207113 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:24.207090 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qkpwq_6616a85b-1ee0-492b-8edc-a50d7d416d9d/routeoverride-cni/0.log" Jul 18 01:39:24.226688 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:24.226648 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qkpwq_6616a85b-1ee0-492b-8edc-a50d7d416d9d/whereabouts-cni-bincopy/0.log" Jul 18 01:39:24.255145 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:24.255123 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qkpwq_6616a85b-1ee0-492b-8edc-a50d7d416d9d/whereabouts-cni/0.log" Jul 18 01:39:24.593341 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:24.593306 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-gh5tf_2336c951-9bb7-46e7-b596-95fca9aa7f6a/kube-multus/0.log" Jul 18 01:39:24.712351 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:24.712315 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kj5ss_e616fcde-1cc6-4b83-b568-b0ac7076a7b3/network-metrics-daemon/0.log" Jul 18 01:39:24.731318 ip-10-0-134-239 kubenswrapper[2580]: I0718 01:39:24.731294 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kj5ss_e616fcde-1cc6-4b83-b568-b0ac7076a7b3/kube-rbac-proxy/0.log"