Jul 07 20:01:18.211173 ip-10-0-133-229 systemd[1]: Starting Kubernetes Kubelet... Jul 07 20:01:21.434793 ip-10-0-133-229 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 20:01:21.434793 ip-10-0-133-229 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 07 20:01:21.434793 ip-10-0-133-229 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 20:01:21.434793 ip-10-0-133-229 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 07 20:01:21.434793 ip-10-0-133-229 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 20:01:21.434793 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.400899 2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 07 20:01:21.434793 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405663 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 20:01:21.434793 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405675 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 20:01:21.434793 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405679 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 20:01:21.434793 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405682 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 20:01:21.434793 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405685 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 20:01:21.434793 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405688 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 20:01:21.434793 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405691 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 20:01:21.434793 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405694 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 20:01:21.436327 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405700 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 20:01:21.436327 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405704 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 20:01:21.436327 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405708 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 20:01:21.436327 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405711 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 20:01:21.436327 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405715 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 20:01:21.436327 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405719 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 20:01:21.436327 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405722 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 20:01:21.436327 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405725 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 20:01:21.436327 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405728 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 20:01:21.436327 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405731 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 20:01:21.436327 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405733 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 20:01:21.436327 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405735 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 20:01:21.436327 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405738 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 20:01:21.436327 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405741 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 20:01:21.436327 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405743 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 20:01:21.436327 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405746 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 20:01:21.436327 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405749 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 20:01:21.436327 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405751 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 20:01:21.438330 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405753 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 20:01:21.438330 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405756 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 20:01:21.438330 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405759 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 20:01:21.438330 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405762 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 20:01:21.438330 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405765 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 20:01:21.438330 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405768 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 20:01:21.438330 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405771 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 20:01:21.438330 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405773 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 20:01:21.438330 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405776 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 20:01:21.438330 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405779 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 20:01:21.438330 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405781 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 20:01:21.438330 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405784 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 20:01:21.438330 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405787 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 20:01:21.438330 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405790 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 20:01:21.438330 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405792 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 20:01:21.438330 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405795 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 20:01:21.438330 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405797 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 20:01:21.438330 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405800 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 20:01:21.438330 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405802 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 20:01:21.438330 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405804 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 20:01:21.439058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405807 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 20:01:21.439058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405810 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 20:01:21.439058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405812 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 20:01:21.439058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405816 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 20:01:21.439058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405818 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 20:01:21.439058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405821 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 20:01:21.439058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405824 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 20:01:21.439058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405827 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 20:01:21.439058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405829 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 20:01:21.439058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405832 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 20:01:21.439058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405834 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 20:01:21.439058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405837 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 20:01:21.439058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405839 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 20:01:21.439058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405842 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 20:01:21.439058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405845 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 20:01:21.439058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405847 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 20:01:21.439058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405850 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 20:01:21.439058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405853 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 20:01:21.439058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405856 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 20:01:21.439058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405859 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 20:01:21.439574 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405862 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 20:01:21.439574 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405865 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 20:01:21.439574 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405867 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 20:01:21.439574 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405870 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 20:01:21.439574 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405872 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 20:01:21.439574 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405875 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 20:01:21.439574 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405878 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 20:01:21.439574 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405880 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 20:01:21.439574 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405883 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 20:01:21.439574 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405885 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 20:01:21.439574 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405888 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 20:01:21.439574 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405891 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 20:01:21.439574 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405894 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 20:01:21.439574 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405897 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 20:01:21.439574 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405899 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 20:01:21.439574 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405902 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 20:01:21.439574 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405904 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 20:01:21.439574 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405908 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 20:01:21.439574 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405910 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 20:01:21.439574 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405914 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 20:01:21.440058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405917 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 20:01:21.440058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405920 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 20:01:21.440058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405923 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 20:01:21.440058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405926 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 20:01:21.440058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405930 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 20:01:21.440058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405932 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 20:01:21.440058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405935 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 07 20:01:21.440058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405938 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 20:01:21.440058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405941 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 20:01:21.440058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405944 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 20:01:21.440058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405947 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 20:01:21.440058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405950 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 20:01:21.440058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.405953 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 20:01:21.440058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406680 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 20:01:21.440058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406687 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 07 20:01:21.440058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406690 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 20:01:21.440058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406692 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 20:01:21.440058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406695 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 20:01:21.440058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406699 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 20:01:21.440058 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406701 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 20:01:21.440575 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406704 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 20:01:21.440575 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406706 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 20:01:21.440575 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406709 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 20:01:21.440575 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406712 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 20:01:21.440575 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406715 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 20:01:21.440575 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406717 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 20:01:21.440575 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406720 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 20:01:21.440575 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406722 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 20:01:21.440575 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406725 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 20:01:21.440575 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406728 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 20:01:21.440575 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406730 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 20:01:21.440575 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406733 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 20:01:21.440575 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406736 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 20:01:21.440575 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406738 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 20:01:21.440575 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406741 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 20:01:21.440575 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406743 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 20:01:21.440575 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406746 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 20:01:21.440575 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406748 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 20:01:21.440575 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406751 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 20:01:21.440575 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406753 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 20:01:21.441053 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406756 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 20:01:21.441053 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406759 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 20:01:21.441053 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406761 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 20:01:21.441053 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406764 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 20:01:21.441053 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406767 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 20:01:21.441053 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406770 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 20:01:21.441053 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406773 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 20:01:21.441053 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406775 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 20:01:21.441053 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406780 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 20:01:21.441053 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406783 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 20:01:21.441053 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406786 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 20:01:21.441053 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406789 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 20:01:21.441053 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406792 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 20:01:21.441053 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406794 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 20:01:21.441053 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406797 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 20:01:21.441053 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406799 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 20:01:21.441053 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406802 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 20:01:21.441053 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406804 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 20:01:21.441053 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406807 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 20:01:21.441053 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406810 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 20:01:21.441552 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406813 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 20:01:21.441552 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406816 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 20:01:21.441552 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406818 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 20:01:21.441552 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406821 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 20:01:21.441552 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406825 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 20:01:21.441552 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406827 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 20:01:21.441552 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406830 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 20:01:21.441552 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406832 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 20:01:21.441552 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406836 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 20:01:21.441552 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406838 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 20:01:21.441552 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406841 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 20:01:21.441552 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406844 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 20:01:21.441552 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406847 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 20:01:21.441552 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406849 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 20:01:21.441552 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406852 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 20:01:21.441552 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406854 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 20:01:21.441552 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406857 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 20:01:21.441552 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406861 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 20:01:21.441552 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406863 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 20:01:21.442015 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406866 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 20:01:21.442015 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406869 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 20:01:21.442015 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406871 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 20:01:21.442015 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406874 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 20:01:21.442015 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406877 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 20:01:21.442015 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406879 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 20:01:21.442015 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406882 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 20:01:21.442015 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406884 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 20:01:21.442015 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406887 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 20:01:21.442015 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406889 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 20:01:21.442015 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406891 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 20:01:21.442015 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406894 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 20:01:21.442015 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406897 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 20:01:21.442015 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406900 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 20:01:21.442015 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406903 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 20:01:21.442015 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406905 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 20:01:21.442015 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406908 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 20:01:21.442015 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406910 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 20:01:21.442015 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406913 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 20:01:21.442015 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406915 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 20:01:21.442515 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406918 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 20:01:21.442515 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406920 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 20:01:21.442515 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406923 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 20:01:21.442515 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406925 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 20:01:21.442515 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406928 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 20:01:21.442515 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406931 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 20:01:21.442515 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406933 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 20:01:21.442515 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406936 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 20:01:21.442515 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406939 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 20:01:21.442515 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406941 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 20:01:21.442515 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406945 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 20:01:21.442515 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406949 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 20:01:21.442515 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.406951 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 20:01:21.442515 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407305 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 20:01:21.442515 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407310 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 20:01:21.442515 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407315 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 20:01:21.442515 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407319 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 20:01:21.442515 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407322 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 20:01:21.442515 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407325 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 20:01:21.442983 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407328 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 20:01:21.442983 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407330 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 20:01:21.442983 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407333 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 20:01:21.442983 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407338 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 20:01:21.442983 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407340 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 20:01:21.442983 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407343 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 20:01:21.442983 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407346 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 20:01:21.442983 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407348 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 20:01:21.442983 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407351 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 20:01:21.442983 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407353 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 20:01:21.442983 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407357 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 20:01:21.442983 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407359 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 20:01:21.442983 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407362 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 20:01:21.442983 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407364 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 20:01:21.442983 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407367 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 07 20:01:21.442983 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407370 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 20:01:21.442983 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407372 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 20:01:21.442983 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407375 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 20:01:21.442983 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407377 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 20:01:21.442983 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407380 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 20:01:21.443476 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407383 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 20:01:21.443476 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407385 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 20:01:21.443476 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407387 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 20:01:21.443476 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407390 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 20:01:21.443476 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407393 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 20:01:21.443476 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407412 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 20:01:21.443476 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407416 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 20:01:21.443476 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407419 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 20:01:21.443476 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407422 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 20:01:21.443476 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407426 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 20:01:21.443476 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407429 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 20:01:21.443476 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407432 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 20:01:21.443476 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407436 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 20:01:21.443476 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407438 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 20:01:21.443476 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407441 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 20:01:21.443476 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407445 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 20:01:21.443476 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407447 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 20:01:21.443476 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407450 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 20:01:21.443476 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407453 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 20:01:21.444037 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407455 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 20:01:21.444037 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407458 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 20:01:21.444037 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407461 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 20:01:21.444037 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407463 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 20:01:21.444037 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407467 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 20:01:21.444037 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407470 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 20:01:21.444037 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407473 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 20:01:21.444037 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407475 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 20:01:21.444037 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407478 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 20:01:21.444037 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407480 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 20:01:21.444037 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407483 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 20:01:21.444037 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407487 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 20:01:21.444037 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407489 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 20:01:21.444037 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407493 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 20:01:21.444037 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407496 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 20:01:21.444037 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407499 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 20:01:21.444037 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407502 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 20:01:21.444037 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407504 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 20:01:21.444037 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407507 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 20:01:21.444590 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407510 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 20:01:21.444590 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407512 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 20:01:21.444590 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407514 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 20:01:21.444590 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407517 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 20:01:21.444590 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407519 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 20:01:21.444590 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407522 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 20:01:21.444590 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407524 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 20:01:21.444590 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407527 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 20:01:21.444590 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407529 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 20:01:21.444590 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407533 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 20:01:21.444590 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407535 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 20:01:21.444590 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407538 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 20:01:21.444590 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407540 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 20:01:21.444590 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407543 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 20:01:21.444590 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407545 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 20:01:21.444590 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407548 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 20:01:21.444590 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407550 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 20:01:21.444590 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407553 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 20:01:21.444590 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407555 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 20:01:21.444590 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407558 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 20:01:21.445128 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407560 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 20:01:21.445128 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407563 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 20:01:21.445128 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407565 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 20:01:21.445128 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407567 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 20:01:21.445128 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407570 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 20:01:21.445128 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407572 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 20:01:21.445128 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407576 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 20:01:21.445128 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407579 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 20:01:21.445128 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407581 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 20:01:21.445128 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407584 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 20:01:21.445128 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407586 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 20:01:21.445128 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407589 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 20:01:21.445128 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407592 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 20:01:21.445128 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407594 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 20:01:21.445128 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.407597 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 20:01:21.445128 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407667 2565 flags.go:64] FLAG: --address="0.0.0.0" Jul 07 20:01:21.445128 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407674 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 07 20:01:21.445128 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407681 2565 flags.go:64] FLAG: --anonymous-auth="true" Jul 07 20:01:21.445128 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407685 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 07 20:01:21.445128 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407689 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 07 20:01:21.445128 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407693 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 07 20:01:21.445670 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407698 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 07 20:01:21.445670 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407703 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 07 20:01:21.445670 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407706 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 07 20:01:21.445670 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407709 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 07 20:01:21.445670 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407712 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 07 20:01:21.445670 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407716 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 07 20:01:21.445670 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407719 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 07 20:01:21.445670 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407722 2565 flags.go:64] FLAG: --cgroup-root="" Jul 07 20:01:21.445670 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407725 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 07 20:01:21.445670 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407728 2565 flags.go:64] FLAG: --client-ca-file="" Jul 07 20:01:21.445670 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407730 2565 flags.go:64] FLAG: --cloud-provider="external" Jul 07 20:01:21.445670 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407733 2565 flags.go:64] FLAG: --cluster-dns="[]" Jul 07 20:01:21.445670 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407737 2565 flags.go:64] FLAG: --cluster-domain="" Jul 07 20:01:21.445670 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407740 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 07 20:01:21.445670 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407743 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 07 20:01:21.445670 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407746 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 07 20:01:21.445670 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407750 2565 flags.go:64] FLAG: --container-log-max-files="5" Jul 07 20:01:21.445670 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407754 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 07 20:01:21.445670 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407757 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 07 20:01:21.445670 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407760 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 07 20:01:21.445670 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407763 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 07 20:01:21.445670 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407767 2565 flags.go:64] FLAG: --contention-profiling="false" Jul 07 20:01:21.445670 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407769 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 07 20:01:21.445670 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407773 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 07 20:01:21.446426 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407779 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 07 20:01:21.446426 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407783 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 07 20:01:21.446426 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407787 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 07 20:01:21.446426 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407790 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 07 20:01:21.446426 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407793 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 07 20:01:21.446426 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407795 2565 flags.go:64] FLAG: --enable-load-reader="false" Jul 07 20:01:21.446426 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407799 2565 flags.go:64] FLAG: --enable-server="true" Jul 07 20:01:21.446426 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407802 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 07 20:01:21.446426 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407808 2565 flags.go:64] FLAG: --event-burst="100" Jul 07 20:01:21.446426 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407812 2565 flags.go:64] FLAG: --event-qps="50" Jul 07 20:01:21.446426 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407814 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 07 20:01:21.446426 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407817 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 07 20:01:21.446426 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407820 2565 flags.go:64] FLAG: --eviction-hard="" Jul 07 20:01:21.446426 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407824 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 07 20:01:21.446426 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407827 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 07 20:01:21.446426 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407830 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 07 20:01:21.446426 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407833 2565 flags.go:64] FLAG: --eviction-soft="" Jul 07 20:01:21.446426 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407835 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 07 20:01:21.446426 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407838 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 07 20:01:21.446426 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407841 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 07 20:01:21.446426 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407844 2565 flags.go:64] FLAG: --experimental-mounter-path="" Jul 07 20:01:21.446426 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407847 2565 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 07 20:01:21.446426 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407850 2565 flags.go:64] FLAG: --fail-swap-on="true" Jul 07 20:01:21.446426 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407853 2565 flags.go:64] FLAG: --feature-gates="" Jul 07 20:01:21.446426 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407857 2565 flags.go:64] FLAG: --file-check-frequency="20s" Jul 07 20:01:21.447307 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407861 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 07 20:01:21.447307 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407864 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 07 20:01:21.447307 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407867 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 07 20:01:21.447307 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407870 2565 flags.go:64] FLAG: --healthz-port="10248" Jul 07 20:01:21.447307 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407873 2565 flags.go:64] FLAG: --help="false" Jul 07 20:01:21.447307 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407876 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-133-229.ec2.internal" Jul 07 20:01:21.447307 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407879 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 07 20:01:21.447307 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407882 2565 flags.go:64] FLAG: --http-check-frequency="20s" Jul 07 20:01:21.447307 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407885 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 07 20:01:21.447307 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407888 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 07 20:01:21.447307 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407892 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 07 20:01:21.447307 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407894 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 07 20:01:21.447307 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407897 2565 flags.go:64] FLAG: --image-service-endpoint="" Jul 07 20:01:21.447307 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407900 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 07 20:01:21.447307 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407903 2565 flags.go:64] FLAG: --kube-api-burst="100" Jul 07 20:01:21.447307 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407907 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 07 20:01:21.447307 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407910 2565 flags.go:64] FLAG: --kube-api-qps="50" Jul 07 20:01:21.447307 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407913 2565 flags.go:64] FLAG: --kube-reserved="" Jul 07 20:01:21.447307 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407916 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 07 20:01:21.447307 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407919 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 07 20:01:21.447307 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407922 2565 flags.go:64] FLAG: --kubelet-cgroups="" Jul 07 20:01:21.447307 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407925 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 07 20:01:21.447307 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407928 2565 flags.go:64] FLAG: --lock-file="" Jul 07 20:01:21.447307 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407930 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 07 20:01:21.448328 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407933 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 07 20:01:21.448328 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407937 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 07 20:01:21.448328 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407942 2565 flags.go:64] FLAG: --log-json-split-stream="false" Jul 07 20:01:21.448328 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407945 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 07 20:01:21.448328 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407948 2565 flags.go:64] FLAG: --log-text-split-stream="false" Jul 07 20:01:21.448328 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407950 2565 flags.go:64] FLAG: --logging-format="text" Jul 07 20:01:21.448328 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407953 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 07 20:01:21.448328 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407957 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 07 20:01:21.448328 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407961 2565 flags.go:64] FLAG: --manifest-url="" Jul 07 20:01:21.448328 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407963 2565 flags.go:64] FLAG: --manifest-url-header="" Jul 07 20:01:21.448328 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407968 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 07 20:01:21.448328 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407971 2565 flags.go:64] FLAG: --max-open-files="1000000" Jul 07 20:01:21.448328 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407974 2565 flags.go:64] FLAG: --max-pods="110" Jul 07 20:01:21.448328 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407977 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 07 20:01:21.448328 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407980 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 07 20:01:21.448328 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407983 2565 flags.go:64] FLAG: --memory-manager-policy="None" Jul 07 20:01:21.448328 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407986 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 07 20:01:21.448328 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407989 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 07 20:01:21.448328 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407992 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 07 20:01:21.448328 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.407995 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 07 20:01:21.448328 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408003 2565 flags.go:64] FLAG: --node-status-max-images="50" Jul 07 20:01:21.448328 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408006 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 07 20:01:21.448328 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408009 2565 flags.go:64] FLAG: --oom-score-adj="-999" Jul 07 20:01:21.448328 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408013 2565 flags.go:64] FLAG: --pod-cidr="" Jul 07 20:01:21.449042 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408016 2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 07 20:01:21.449042 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408022 2565 flags.go:64] FLAG: --pod-manifest-path="" Jul 07 20:01:21.449042 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408024 2565 flags.go:64] FLAG: --pod-max-pids="-1" Jul 07 20:01:21.449042 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408027 2565 flags.go:64] FLAG: --pods-per-core="0" Jul 07 20:01:21.449042 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408030 2565 flags.go:64] FLAG: --port="10250" Jul 07 20:01:21.449042 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408033 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 07 20:01:21.449042 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408036 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02870c378b0fb7868" Jul 07 20:01:21.449042 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408039 2565 flags.go:64] FLAG: --qos-reserved="" Jul 07 20:01:21.449042 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408042 2565 flags.go:64] FLAG: --read-only-port="10255" Jul 07 20:01:21.449042 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408045 2565 flags.go:64] FLAG: --register-node="true" Jul 07 20:01:21.449042 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408048 2565 flags.go:64] FLAG: --register-with-taints="" Jul 07 20:01:21.449042 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408051 2565 flags.go:64] FLAG: --registry-burst="10" Jul 07 20:01:21.449042 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408054 2565 flags.go:64] FLAG: --registry-qps="5" Jul 07 20:01:21.449042 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408057 2565 flags.go:64] FLAG: --reserved-cpus="" Jul 07 20:01:21.449042 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408059 2565 flags.go:64] FLAG: --reserved-memory="" Jul 07 20:01:21.449042 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408063 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 07 20:01:21.449042 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408068 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 07 20:01:21.449042 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408071 2565 flags.go:64] FLAG: --rotate-certificates="false" Jul 07 20:01:21.449042 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408074 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 07 20:01:21.449042 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408077 2565 flags.go:64] FLAG: --runonce="false" Jul 07 20:01:21.449042 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408080 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 07 20:01:21.449042 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408083 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 07 20:01:21.449042 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408086 2565 flags.go:64] FLAG: --seccomp-default="false" Jul 07 20:01:21.449042 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408089 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 07 20:01:21.449042 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408092 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 07 20:01:21.449652 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408095 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 07 20:01:21.449652 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408098 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 07 20:01:21.449652 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408101 2565 flags.go:64] FLAG: --storage-driver-password="root" Jul 07 20:01:21.449652 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408104 2565 flags.go:64] FLAG: --storage-driver-secure="false" Jul 07 20:01:21.449652 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408107 2565 flags.go:64] FLAG: --storage-driver-table="stats" Jul 07 20:01:21.449652 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408110 2565 flags.go:64] FLAG: --storage-driver-user="root" Jul 07 20:01:21.449652 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408112 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 07 20:01:21.449652 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408116 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 07 20:01:21.449652 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408119 2565 flags.go:64] FLAG: --system-cgroups="" Jul 07 20:01:21.449652 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408122 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 07 20:01:21.449652 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408127 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 07 20:01:21.449652 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408130 2565 flags.go:64] FLAG: --tls-cert-file="" Jul 07 20:01:21.449652 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408133 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 07 20:01:21.449652 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408136 2565 flags.go:64] FLAG: --tls-min-version="" Jul 07 20:01:21.449652 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408139 2565 flags.go:64] FLAG: --tls-private-key-file="" Jul 07 20:01:21.449652 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408142 2565 flags.go:64] FLAG: --topology-manager-policy="none" Jul 07 20:01:21.449652 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408145 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 07 20:01:21.449652 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408147 2565 flags.go:64] FLAG: --topology-manager-scope="container" Jul 07 20:01:21.449652 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408150 2565 flags.go:64] FLAG: --v="2" Jul 07 20:01:21.449652 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408154 2565 flags.go:64] FLAG: --version="false" Jul 07 20:01:21.449652 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408158 2565 flags.go:64] FLAG: --vmodule="" Jul 07 20:01:21.449652 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408165 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 07 20:01:21.449652 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.408169 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 07 20:01:21.449652 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408284 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 20:01:21.449652 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408288 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 20:01:21.450263 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408291 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 20:01:21.450263 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408294 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 20:01:21.450263 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408296 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 20:01:21.450263 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408300 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 20:01:21.450263 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408303 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 20:01:21.450263 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408306 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 20:01:21.450263 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408308 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 20:01:21.450263 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408311 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 20:01:21.450263 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408314 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 20:01:21.450263 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408317 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 20:01:21.450263 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408320 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 20:01:21.450263 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408322 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 20:01:21.450263 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408325 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 20:01:21.450263 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408327 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 20:01:21.450263 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408331 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 20:01:21.450263 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408334 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 20:01:21.450263 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408336 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 20:01:21.450263 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408339 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 20:01:21.450263 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408341 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 20:01:21.450263 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408344 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 20:01:21.450263 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408347 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 20:01:21.450863 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408349 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 20:01:21.450863 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408352 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 20:01:21.450863 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408354 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 20:01:21.450863 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408357 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 20:01:21.450863 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408359 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 20:01:21.450863 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408362 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 20:01:21.450863 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408364 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 20:01:21.450863 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408367 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 20:01:21.450863 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408369 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 20:01:21.450863 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408373 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 20:01:21.450863 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408375 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 20:01:21.450863 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408378 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 20:01:21.450863 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408380 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 20:01:21.450863 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408383 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 20:01:21.450863 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408385 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 20:01:21.450863 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408387 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 20:01:21.450863 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408390 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 20:01:21.450863 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408392 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 20:01:21.450863 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408409 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 20:01:21.450863 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408413 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 20:01:21.451331 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408416 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 20:01:21.451331 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408419 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 20:01:21.451331 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408422 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 20:01:21.451331 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408425 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 20:01:21.451331 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408427 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 20:01:21.451331 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408431 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 20:01:21.451331 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408435 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 20:01:21.451331 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408439 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 20:01:21.451331 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408442 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 20:01:21.451331 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408445 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 20:01:21.451331 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408447 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 20:01:21.451331 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408450 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 20:01:21.451331 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408452 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 20:01:21.451331 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408455 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 20:01:21.451331 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408457 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 20:01:21.451331 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408460 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 20:01:21.451331 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408463 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 20:01:21.451331 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408465 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 20:01:21.451331 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408468 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 20:01:21.451803 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408470 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 20:01:21.451803 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408474 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 20:01:21.451803 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408479 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 20:01:21.451803 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408481 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 20:01:21.451803 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408484 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 20:01:21.451803 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408486 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 20:01:21.451803 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408489 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 20:01:21.451803 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408491 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 20:01:21.451803 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408494 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 20:01:21.451803 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408496 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 20:01:21.451803 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408499 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 20:01:21.451803 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408502 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 20:01:21.451803 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408506 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 20:01:21.451803 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408509 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 20:01:21.451803 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408511 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 07 20:01:21.451803 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408515 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 20:01:21.451803 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408517 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 20:01:21.451803 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408520 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 20:01:21.451803 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408524 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 20:01:21.452247 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408527 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 20:01:21.452247 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408529 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 20:01:21.452247 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408532 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 20:01:21.452247 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408534 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 20:01:21.452247 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408537 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 20:01:21.452247 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408539 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 20:01:21.452247 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408542 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 20:01:21.452247 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408545 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 20:01:21.452247 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408548 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 20:01:21.452247 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408550 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 20:01:21.452247 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408560 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 20:01:21.452247 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408563 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 20:01:21.452247 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408566 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 20:01:21.452247 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408569 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 20:01:21.452247 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408571 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 20:01:21.452247 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408575 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 20:01:21.452247 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408578 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 20:01:21.452247 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.408580 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 20:01:21.452706 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.409252 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 20:01:21.452706 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.416832 2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 07 20:01:21.452706 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.416849 2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 07 20:01:21.452706 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.417704 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 07 20:01:21.452706 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.417713 2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 07 20:01:21.452706 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420055 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 20:01:21.452706 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420071 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 20:01:21.452706 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420075 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 20:01:21.452706 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420078 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 20:01:21.452706 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420081 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 20:01:21.452706 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420084 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 20:01:21.452706 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420090 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 20:01:21.452706 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420093 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 20:01:21.452706 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420097 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 20:01:21.453137 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420198 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 20:01:21.453137 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420205 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 20:01:21.453137 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420208 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 20:01:21.453137 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420210 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 20:01:21.453137 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420213 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 20:01:21.453137 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420216 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 20:01:21.453137 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420219 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 20:01:21.453137 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420222 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 20:01:21.453137 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420225 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 20:01:21.453137 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420228 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 20:01:21.453137 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420230 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 20:01:21.453137 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420233 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 20:01:21.453137 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420235 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 20:01:21.453137 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420238 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 20:01:21.453137 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420241 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 20:01:21.453137 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420244 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 20:01:21.453137 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420246 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 20:01:21.453137 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420249 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 20:01:21.453137 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420252 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 20:01:21.453137 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420255 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 20:01:21.453642 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420257 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 20:01:21.453642 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420260 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 20:01:21.453642 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420262 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 20:01:21.453642 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420265 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 20:01:21.453642 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420271 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 20:01:21.453642 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420275 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 20:01:21.453642 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420278 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 20:01:21.453642 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420280 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 20:01:21.453642 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420283 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 20:01:21.453642 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420286 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 20:01:21.453642 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420288 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 20:01:21.453642 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420291 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 07 20:01:21.453642 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420294 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 20:01:21.453642 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420296 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 20:01:21.453642 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420299 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 20:01:21.453642 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420302 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 20:01:21.453642 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420306 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 20:01:21.453642 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420311 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 20:01:21.453642 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420314 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 20:01:21.453642 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420316 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 20:01:21.454153 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420320 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 20:01:21.454153 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420324 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 20:01:21.454153 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420327 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 20:01:21.454153 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420330 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 20:01:21.454153 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420333 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 20:01:21.454153 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420336 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 20:01:21.454153 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420339 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 20:01:21.454153 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420342 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 20:01:21.454153 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420346 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 20:01:21.454153 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420349 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 20:01:21.454153 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420352 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 20:01:21.454153 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420355 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 20:01:21.454153 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420358 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 20:01:21.454153 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420361 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 20:01:21.454153 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420364 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 20:01:21.454153 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420367 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 20:01:21.454153 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420374 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 20:01:21.454153 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420379 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 20:01:21.454153 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420381 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 20:01:21.454153 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420386 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 20:01:21.454647 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420388 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 20:01:21.454647 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420393 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 20:01:21.454647 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420410 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 20:01:21.454647 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420414 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 20:01:21.454647 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420418 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 20:01:21.454647 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420421 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 20:01:21.454647 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420423 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 20:01:21.454647 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420426 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 20:01:21.454647 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420428 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 20:01:21.454647 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420431 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 20:01:21.454647 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420434 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 20:01:21.454647 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420436 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 20:01:21.454647 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420439 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 20:01:21.454647 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420442 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 20:01:21.454647 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420444 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 20:01:21.454647 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420447 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 20:01:21.454647 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420449 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 20:01:21.454647 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420452 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 20:01:21.454647 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420455 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 20:01:21.454647 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420458 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 20:01:21.455117 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420460 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 20:01:21.455117 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420463 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 20:01:21.455117 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420467 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 20:01:21.455117 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420470 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 20:01:21.455117 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420473 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 20:01:21.455117 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420476 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 20:01:21.455117 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420479 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 20:01:21.455117 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420482 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 20:01:21.455117 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420487 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 20:01:21.455117 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420490 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 20:01:21.455117 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.420496 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 20:01:21.455117 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420638 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 20:01:21.455117 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420643 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 20:01:21.455117 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420646 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 20:01:21.455117 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420649 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 07 20:01:21.455488 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420651 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 20:01:21.455488 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420654 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 20:01:21.455488 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420657 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 20:01:21.455488 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420659 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 20:01:21.455488 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420662 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 20:01:21.455488 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420664 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 20:01:21.455488 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420667 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 20:01:21.455488 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420670 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 20:01:21.455488 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420672 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 20:01:21.455488 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420675 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 20:01:21.455488 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420678 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 20:01:21.455488 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420680 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 20:01:21.455488 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420684 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 20:01:21.455488 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420686 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 20:01:21.455488 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420689 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 20:01:21.455488 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420692 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 20:01:21.455488 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420694 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 20:01:21.455488 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420696 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 20:01:21.455488 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420699 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 20:01:21.455939 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420701 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 20:01:21.455939 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420704 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 20:01:21.455939 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420706 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 20:01:21.455939 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420709 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 20:01:21.455939 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420711 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 20:01:21.455939 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420714 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 20:01:21.455939 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420718 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 20:01:21.455939 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420721 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 20:01:21.455939 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420724 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 20:01:21.455939 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420726 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 20:01:21.455939 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420729 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 20:01:21.455939 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420732 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 20:01:21.455939 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420734 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 20:01:21.455939 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420737 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 20:01:21.455939 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420739 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 20:01:21.455939 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420742 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 20:01:21.455939 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420744 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 20:01:21.455939 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420747 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 20:01:21.455939 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420749 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 20:01:21.455939 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420751 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 20:01:21.456456 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420754 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 20:01:21.456456 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420756 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 20:01:21.456456 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420759 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 20:01:21.456456 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420761 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 20:01:21.456456 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420764 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 20:01:21.456456 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420766 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 20:01:21.456456 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420770 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 20:01:21.456456 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420772 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 20:01:21.456456 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420775 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 20:01:21.456456 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420779 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 20:01:21.456456 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420782 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 20:01:21.456456 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420784 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 20:01:21.456456 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420787 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 20:01:21.456456 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420790 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 20:01:21.456456 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420792 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 20:01:21.456456 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420795 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 20:01:21.456456 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420798 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 20:01:21.456456 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420800 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 20:01:21.456456 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420804 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 20:01:21.456987 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420807 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 20:01:21.456987 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420810 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 20:01:21.456987 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420812 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 20:01:21.456987 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420815 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 20:01:21.456987 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420818 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 20:01:21.456987 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420820 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 20:01:21.456987 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420823 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 20:01:21.456987 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420825 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 20:01:21.456987 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420828 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 20:01:21.456987 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420830 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 20:01:21.456987 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420833 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 20:01:21.456987 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420835 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 20:01:21.456987 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420837 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 20:01:21.456987 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420840 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 20:01:21.456987 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420843 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 20:01:21.456987 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420845 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 20:01:21.456987 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420848 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 20:01:21.456987 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420850 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 20:01:21.456987 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420853 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 20:01:21.457464 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420857 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 20:01:21.457464 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420860 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 20:01:21.457464 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420863 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 20:01:21.457464 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420866 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 20:01:21.457464 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420868 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 20:01:21.457464 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420871 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 20:01:21.457464 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420873 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 20:01:21.457464 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420876 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 20:01:21.457464 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420879 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 20:01:21.457464 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420881 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 20:01:21.457464 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420884 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 20:01:21.457464 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420886 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 20:01:21.457464 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420891 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 20:01:21.457464 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420896 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 20:01:21.457464 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420898 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 20:01:21.457464 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420901 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 20:01:21.457464 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420904 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 20:01:21.457464 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:21.420906 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 20:01:21.457901 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.420912 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 20:01:21.457901 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.421012 2565 server.go:962] "Client rotation is on, will bootstrap in background" Jul 07 20:01:21.457901 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.423130 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 07 20:01:21.457901 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.424734 2565 server.go:1019] "Starting client certificate rotation" Jul 07 20:01:21.457901 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.424831 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 20:01:21.457901 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.425623 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 20:01:21.457901 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.427544 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 20:01:21.457901 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.428708 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 20:01:21.457901 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.443555 2565 log.go:25] "Validated CRI v1 runtime API" Jul 07 20:01:21.457901 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.445879 2565 log.go:25] "Validated CRI v1 image API" Jul 07 20:01:21.457901 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.447429 2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 07 20:01:21.457901 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.450125 2565 fs.go:135] Filesystem UUIDs: map[09985dd2-9323-4179-be96-1175a378ce01:/dev/nvme0n1p4 2d1e99f7-7934-4012-ac8e-1c8fff71471b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 07 20:01:21.458205 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.450141 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 07 20:01:21.458205 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.456573 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 20:01:21.458264 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.456591 2565 manager.go:216] Machine: {Timestamp:2026-07-07 20:01:21.455274571 +0000 UTC m=+1.609736510 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101119 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec254acb2219e399b688206b5152cc51 SystemUUID:ec254acb-2219-e399-b688-206b5152cc51 BootID:ef0ecc0d-d308-40e5-8288-9b8121ff2c31 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:52:a6:5c:19:bf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:52:a6:5c:19:bf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:f2:93:75:d7:c4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 07 20:01:21.458264 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.456710 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 07 20:01:21.458264 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.456785 2565 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 07 20:01:21.458264 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.457861 2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 07 20:01:21.458264 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.457881 2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-229.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 07 20:01:21.458553 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.458542 2565 topology_manager.go:138] "Creating topology manager with none policy" Jul 07 20:01:21.458587 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.458554 2565 container_manager_linux.go:309] "Creating device plugin manager" Jul 07 20:01:21.458587 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.458562 2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 20:01:21.459196 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.459185 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 20:01:21.460595 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.460585 2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 07 20:01:21.463568 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.463557 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 07 20:01:21.463671 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.463662 2565 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 07 20:01:21.463701 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.463691 2565 kubelet.go:486] "Attempting to sync node with API server" Jul 07 20:01:21.463731 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.463701 2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 07 20:01:21.463731 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.463715 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 07 20:01:21.463731 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.463724 2565 kubelet.go:398] "Adding apiserver pod source" Jul 07 20:01:21.463823 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.463732 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 07 20:01:21.464257 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.464237 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 20:01:21.464370 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.464269 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 20:01:21.467908 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.467884 2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 07 20:01:21.469995 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.469981 2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 07 20:01:21.470050 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.470012 2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 07 20:01:21.471128 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.471115 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 07 20:01:21.471186 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.471132 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 07 20:01:21.471186 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.471139 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 07 20:01:21.471186 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.471144 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 07 20:01:21.471186 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.471153 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 07 20:01:21.471186 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.471161 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 07 20:01:21.471186 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.471171 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 07 20:01:21.471186 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.471175 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 07 20:01:21.471186 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.471182 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 07 20:01:21.471186 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.471187 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 07 20:01:21.471445 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.471195 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 07 20:01:21.471445 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.471200 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 07 20:01:21.471815 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.471804 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 07 20:01:21.471844 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.471815 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 07 20:01:21.475267 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.475252 2565 server.go:1268] "Started kubelet" Jul 07 20:01:21.475374 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.475326 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 07 20:01:21.475457 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.475345 2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 07 20:01:21.475596 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.475412 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 07 20:01:21.476173 ip-10-0-133-229 systemd[1]: Started Kubernetes Kubelet. Jul 07 20:01:21.476647 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:21.476192 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 07 20:01:21.476647 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.476226 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-229.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 07 20:01:21.476647 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:21.476355 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-229.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 07 20:01:21.476934 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.476867 2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 07 20:01:21.477882 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.477868 2565 server.go:310] "Adding debug handlers to kubelet server" Jul 07 20:01:21.480913 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:21.480885 2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 07 20:01:21.484753 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.484729 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 07 20:01:21.484753 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.484739 2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 07 20:01:21.484854 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:21.484831 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-229.ec2.internal\" not found" Jul 07 20:01:21.485494 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.485470 2565 factory.go:55] Registering systemd factory Jul 07 20:01:21.485570 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.485513 2565 factory.go:223] Registration of the systemd container factory successfully Jul 07 20:01:21.485619 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.485588 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 07 20:01:21.485619 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.485607 2565 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 07 20:01:21.485706 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.485622 2565 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 07 20:01:21.485791 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.485773 2565 reconstruct.go:97] "Volume reconstruction finished" Jul 07 20:01:21.485791 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.485791 2565 reconciler.go:29] "Reconciler: start to sync state" Jul 07 20:01:21.485905 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.485807 2565 factory.go:153] Registering CRI-O factory Jul 07 20:01:21.485905 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.485822 2565 factory.go:223] Registration of the crio container factory successfully Jul 07 20:01:21.486437 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.485874 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 07 20:01:21.486504 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.486454 2565 factory.go:103] Registering Raw factory Jul 07 20:01:21.486504 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.486473 2565 manager.go:1196] Started watching for new ooms in manager Jul 07 20:01:21.486884 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.486867 2565 manager.go:349] Starting recovery of all containers Jul 07 20:01:21.487498 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:21.486250 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-229.ec2.internal.18c01a66d968c79f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-229.ec2.internal,UID:ip-10-0-133-229.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-229.ec2.internal,},FirstTimestamp:2026-07-07 20:01:21.475225503 +0000 UTC m=+1.629687443,LastTimestamp:2026-07-07 20:01:21.475225503 +0000 UTC m=+1.629687443,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-229.ec2.internal,}" Jul 07 20:01:21.491875 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:21.491842 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 07 20:01:21.492115 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:21.492086 2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-229.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 07 20:01:21.496988 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.496955 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 07 20:01:21.498264 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.498244 2565 manager.go:354] Recovery completed Jul 07 20:01:21.498557 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.498536 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8fl5p" Jul 07 20:01:21.503131 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.503117 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 20:01:21.504787 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.504771 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-229.ec2.internal" event="NodeHasSufficientMemory" Jul 07 20:01:21.504851 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.504802 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-229.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 20:01:21.504851 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.504814 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-229.ec2.internal" event="NodeHasSufficientPID" Jul 07 20:01:21.505286 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.505271 2565 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 07 20:01:21.505286 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.505283 2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 07 20:01:21.505441 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.505298 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 07 20:01:21.505441 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.505285 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 20:01:21.506010 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.505996 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-229.ec2.internal" event="NodeHasSufficientMemory" Jul 07 20:01:21.506076 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.506024 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-229.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 20:01:21.506076 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.506038 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-229.ec2.internal" event="NodeHasSufficientPID" Jul 07 20:01:21.506935 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:21.506876 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-229.ec2.internal.18c01a66db2bd8ec default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-229.ec2.internal,UID:ip-10-0-133-229.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-229.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-229.ec2.internal,},FirstTimestamp:2026-07-07 20:01:21.504786668 +0000 UTC m=+1.659248608,LastTimestamp:2026-07-07 20:01:21.504786668 +0000 UTC m=+1.659248608,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-229.ec2.internal,}" Jul 07 20:01:21.507369 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.507358 2565 policy_none.go:49] "None policy: Start" Jul 07 20:01:21.507429 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.507378 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 07 20:01:21.507429 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.507390 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 07 20:01:21.507954 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.507940 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8fl5p" Jul 07 20:01:21.509408 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.509386 2565 policy_none.go:47] "Start" Jul 07 20:01:21.542102 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.541979 2565 manager.go:337] "Starting Device Plugin manager" Jul 07 20:01:21.542184 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:21.542122 2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 07 20:01:21.542184 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.542135 2565 server.go:87] "Starting device plugin registration server" Jul 07 20:01:21.542490 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.542476 2565 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 07 20:01:21.542545 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.542491 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 07 20:01:21.542592 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.542578 2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 07 20:01:21.542667 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.542653 2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 07 20:01:21.542667 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.542665 2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 07 20:01:21.543149 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:21.543075 2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 07 20:01:21.543149 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:21.543131 2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-229.ec2.internal\" not found" Jul 07 20:01:21.642872 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.642847 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 20:01:21.644086 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.643970 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-229.ec2.internal" event="NodeHasSufficientMemory" Jul 07 20:01:21.644086 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.644003 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-229.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 20:01:21.644086 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.644015 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-229.ec2.internal" event="NodeHasSufficientPID" Jul 07 20:01:21.644086 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.644045 2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-229.ec2.internal" Jul 07 20:01:21.653038 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.653018 2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-229.ec2.internal" Jul 07 20:01:21.653038 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:21.653038 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-229.ec2.internal\": node \"ip-10-0-133-229.ec2.internal\" not found" Jul 07 20:01:21.670065 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:21.670044 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-229.ec2.internal\" not found" Jul 07 20:01:21.678317 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.678295 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 07 20:01:21.678412 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.678341 2565 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 07 20:01:21.678412 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.678366 2565 kubelet.go:2448] "Starting kubelet main sync loop" Jul 07 20:01:21.678520 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:21.678416 2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 07 20:01:21.680596 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.680581 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 20:01:21.770768 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:21.770720 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-229.ec2.internal\" not found" Jul 07 20:01:21.778880 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.778862 2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-229.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-229.ec2.internal"] Jul 07 20:01:21.778945 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.778935 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 20:01:21.780450 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.780436 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-229.ec2.internal" event="NodeHasSufficientMemory" Jul 07 20:01:21.780520 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.780463 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-229.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 20:01:21.780520 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.780472 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-229.ec2.internal" event="NodeHasSufficientPID" Jul 07 20:01:21.781900 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.781886 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 20:01:21.782014 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.782000 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-229.ec2.internal" Jul 07 20:01:21.782051 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.782029 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 20:01:21.782637 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.782620 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-229.ec2.internal" event="NodeHasSufficientMemory" Jul 07 20:01:21.782719 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.782647 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-229.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 20:01:21.782719 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.782621 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-229.ec2.internal" event="NodeHasSufficientMemory" Jul 07 20:01:21.782719 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.782662 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-229.ec2.internal" event="NodeHasSufficientPID" Jul 07 20:01:21.782719 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.782677 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-229.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 20:01:21.782719 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.782689 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-229.ec2.internal" event="NodeHasSufficientPID" Jul 07 20:01:21.783851 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.783838 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-229.ec2.internal" Jul 07 20:01:21.783919 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.783861 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 20:01:21.785324 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.785310 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-229.ec2.internal" event="NodeHasSufficientMemory" Jul 07 20:01:21.785420 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.785335 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-229.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 20:01:21.785420 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.785346 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-229.ec2.internal" event="NodeHasSufficientPID" Jul 07 20:01:21.787465 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.787449 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/59bf4ed301a47b79107844633c8ed6ad-config\") pod \"kube-apiserver-proxy-ip-10-0-133-229.ec2.internal\" (UID: \"59bf4ed301a47b79107844633c8ed6ad\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-229.ec2.internal" Jul 07 20:01:21.787539 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.787471 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/da4ff7f5b4692b4311b38219eb74d174-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-229.ec2.internal\" (UID: \"da4ff7f5b4692b4311b38219eb74d174\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-229.ec2.internal" Jul 07 20:01:21.787539 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.787491 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da4ff7f5b4692b4311b38219eb74d174-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-229.ec2.internal\" (UID: \"da4ff7f5b4692b4311b38219eb74d174\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-229.ec2.internal" Jul 07 20:01:21.810061 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:21.810034 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-229.ec2.internal\" not found" node="ip-10-0-133-229.ec2.internal" Jul 07 20:01:21.813707 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:21.813688 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-229.ec2.internal\" not found" node="ip-10-0-133-229.ec2.internal" Jul 07 20:01:21.871464 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:21.871444 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-229.ec2.internal\" not found" Jul 07 20:01:21.887694 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.887672 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/da4ff7f5b4692b4311b38219eb74d174-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-229.ec2.internal\" (UID: \"da4ff7f5b4692b4311b38219eb74d174\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-229.ec2.internal" Jul 07 20:01:21.887772 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.887695 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da4ff7f5b4692b4311b38219eb74d174-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-229.ec2.internal\" (UID: \"da4ff7f5b4692b4311b38219eb74d174\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-229.ec2.internal" Jul 07 20:01:21.887772 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.887711 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/59bf4ed301a47b79107844633c8ed6ad-config\") pod \"kube-apiserver-proxy-ip-10-0-133-229.ec2.internal\" (UID: \"59bf4ed301a47b79107844633c8ed6ad\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-229.ec2.internal" Jul 07 20:01:21.887772 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.887737 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/59bf4ed301a47b79107844633c8ed6ad-config\") pod \"kube-apiserver-proxy-ip-10-0-133-229.ec2.internal\" (UID: \"59bf4ed301a47b79107844633c8ed6ad\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-229.ec2.internal" Jul 07 20:01:21.887896 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.887780 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da4ff7f5b4692b4311b38219eb74d174-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-229.ec2.internal\" (UID: \"da4ff7f5b4692b4311b38219eb74d174\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-229.ec2.internal" Jul 07 20:01:21.887896 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:21.887781 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/da4ff7f5b4692b4311b38219eb74d174-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-229.ec2.internal\" (UID: \"da4ff7f5b4692b4311b38219eb74d174\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-229.ec2.internal" Jul 07 20:01:21.971856 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:21.971824 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-229.ec2.internal\" not found" Jul 07 20:01:22.072344 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:22.072280 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-229.ec2.internal\" not found" Jul 07 20:01:22.112829 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:22.112800 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-229.ec2.internal" Jul 07 20:01:22.116934 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:22.116812 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-229.ec2.internal" Jul 07 20:01:22.173121 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:22.173096 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-229.ec2.internal\" not found" Jul 07 20:01:22.273684 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:22.273655 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-229.ec2.internal\" not found" Jul 07 20:01:22.374176 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:22.374107 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-229.ec2.internal\" not found" Jul 07 20:01:22.400018 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:22.399987 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 20:01:22.424258 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:22.424228 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 07 20:01:22.424385 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:22.424362 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 20:01:22.424385 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:22.424364 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 20:01:22.474640 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:22.474621 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-229.ec2.internal\" not found" Jul 07 20:01:22.510911 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:22.510881 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 19:56:21 +0000 UTC" deadline="2028-03-09 19:18:43.221708443 +0000 UTC" Jul 07 20:01:22.510911 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:22.510904 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14663h17m20.710807286s" Jul 07 20:01:22.575219 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:22.575172 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-229.ec2.internal\" not found" Jul 07 20:01:22.626924 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:22.626850 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 20:01:22.675649 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:22.675621 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-229.ec2.internal\" not found" Jul 07 20:01:22.688605 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:22.688584 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 20:01:22.703087 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:22.703057 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4ff7f5b4692b4311b38219eb74d174.slice/crio-ec9c5e0e4d0ea04ce2faa75ab7d747e184146c533717ad88fbe0a704a7074096 WatchSource:0}: Error finding container ec9c5e0e4d0ea04ce2faa75ab7d747e184146c533717ad88fbe0a704a7074096: Status 404 returned error can't find the container with id ec9c5e0e4d0ea04ce2faa75ab7d747e184146c533717ad88fbe0a704a7074096 Jul 07 20:01:22.707157 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:22.707145 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 20:01:22.777887 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:22.777850 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59bf4ed301a47b79107844633c8ed6ad.slice/crio-2ee03d5f441c0b4e8c2d01a947ede35539f0134c600f55a4e27d66cd747bd764 WatchSource:0}: Error finding container 2ee03d5f441c0b4e8c2d01a947ede35539f0134c600f55a4e27d66cd747bd764: Status 404 returned error can't find the container with id 2ee03d5f441c0b4e8c2d01a947ede35539f0134c600f55a4e27d66cd747bd764 Jul 07 20:01:22.785768 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:22.785747 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-229.ec2.internal" Jul 07 20:01:22.807388 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:22.807371 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 20:01:22.808450 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:22.808438 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-229.ec2.internal" Jul 07 20:01:22.821427 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:22.821415 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 20:01:23.465673 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.465629 2565 apiserver.go:52] "Watching apiserver" Jul 07 20:01:23.475120 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.475092 2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 07 20:01:23.475855 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.475832 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-229.ec2.internal","openshift-multus/multus-25ms9","openshift-multus/multus-additional-cni-plugins-vx82p","openshift-multus/network-metrics-daemon-qvph6","openshift-ovn-kubernetes/ovnkube-node-6skbt","kube-system/konnectivity-agent-jlj7d","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfbfw","openshift-cluster-node-tuning-operator/tuned-7gbnw","openshift-dns/node-resolver-b8qtm","openshift-image-registry/node-ca-jbxhj","openshift-network-diagnostics/network-check-target-wj7w5","openshift-network-operator/iptables-alerter-v6vf5","kube-system/global-pull-secret-syncer-zr7c2","kube-system/kube-apiserver-proxy-ip-10-0-133-229.ec2.internal"] Jul 07 20:01:23.477385 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.477366 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfbfw" Jul 07 20:01:23.478745 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.478725 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:01:23.478828 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:23.478799 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wj7w5" podUID="92ea0d19-f83c-4b0d-969d-cf288f52c55a" Jul 07 20:01:23.480111 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.480080 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-blhsp\"" Jul 07 20:01:23.480214 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.480158 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 07 20:01:23.480286 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.480270 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 07 20:01:23.480420 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.480384 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 07 20:01:23.480517 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.480420 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vx82p" Jul 07 20:01:23.481689 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.481667 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:01:23.481785 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:23.481728 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvph6" podUID="da37b4bb-9b02-4491-9d87-c4f56be7c7ad" Jul 07 20:01:23.483348 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.483220 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 07 20:01:23.483348 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.483257 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 07 20:01:23.483348 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.483224 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 07 20:01:23.483696 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.483587 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 07 20:01:23.483696 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.483605 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 07 20:01:23.483838 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.483823 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.483838 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.483830 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k7lct\"" Jul 07 20:01:23.485166 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.485146 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jlj7d" Jul 07 20:01:23.485701 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.485684 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 07 20:01:23.486802 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.486536 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b8qtm" Jul 07 20:01:23.486802 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.486576 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 07 20:01:23.486802 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.486600 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 07 20:01:23.486802 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.486538 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rlrgc\"" Jul 07 20:01:23.487690 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.487675 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 07 20:01:23.487984 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.487961 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v6vf5" Jul 07 20:01:23.488195 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.488140 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 07 20:01:23.489488 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.489307 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hl2ps\"" Jul 07 20:01:23.489488 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.489364 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7fqh6\"" Jul 07 20:01:23.489645 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.489631 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.491169 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.491152 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jbxhj" Jul 07 20:01:23.492010 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.491875 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 07 20:01:23.492309 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.492140 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 07 20:01:23.492309 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.492181 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 07 20:01:23.492309 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.492288 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 07 20:01:23.492499 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.492368 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 07 20:01:23.492670 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.492642 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 07 20:01:23.492670 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.492650 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 07 20:01:23.493354 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.493336 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 07 20:01:23.493354 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.493348 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.493565 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.493471 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6v9cd\"" Jul 07 20:01:23.493565 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.493479 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 07 20:01:23.493565 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.493551 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 07 20:01:23.493827 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.493813 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4gdht\"" Jul 07 20:01:23.493902 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.493818 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 07 20:01:23.494593 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.494578 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 07 20:01:23.495171 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.495153 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:23.495260 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:23.495223 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr7c2" podUID="17142129-e574-4092-bd8d-0ffa3de6507a" Jul 07 20:01:23.495319 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.495283 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 07 20:01:23.495387 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.495370 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-sys\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.495464 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.495421 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vx82p\" (UID: \"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39\") " pod="openshift-multus/multus-additional-cni-plugins-vx82p" Jul 07 20:01:23.495464 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.495452 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-etc-sysctl-conf\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.495560 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.495464 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 07 20:01:23.495560 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.495532 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-lib-modules\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.495657 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.495587 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-host\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.495657 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.495615 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2c31e770-b420-42cf-8bf6-a7f8540ed592-agent-certs\") pod \"konnectivity-agent-jlj7d\" (UID: \"2c31e770-b420-42cf-8bf6-a7f8540ed592\") " pod="kube-system/konnectivity-agent-jlj7d" Jul 07 20:01:23.495747 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.495651 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vx82p\" (UID: \"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39\") " pod="openshift-multus/multus-additional-cni-plugins-vx82p" Jul 07 20:01:23.495747 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.495740 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-etc-sysconfig\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.495830 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.495763 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-etc-sysctl-d\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.495830 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.495799 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-etc-systemd\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.495893 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.495852 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwtp2\" (UniqueName: \"kubernetes.io/projected/19ef0176-73b8-465c-abda-754d10ddd21f-kube-api-access-nwtp2\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.495893 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.495878 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc753a68-481d-469b-88a3-37bbb48ac522-registration-dir\") pod \"aws-ebs-csi-driver-node-dfbfw\" (UID: \"dc753a68-481d-469b-88a3-37bbb48ac522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfbfw" Jul 07 20:01:23.495967 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.495915 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc753a68-481d-469b-88a3-37bbb48ac522-device-dir\") pod \"aws-ebs-csi-driver-node-dfbfw\" (UID: \"dc753a68-481d-469b-88a3-37bbb48ac522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfbfw" Jul 07 20:01:23.495967 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.495962 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc753a68-481d-469b-88a3-37bbb48ac522-sys-fs\") pod \"aws-ebs-csi-driver-node-dfbfw\" (UID: \"dc753a68-481d-469b-88a3-37bbb48ac522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfbfw" Jul 07 20:01:23.496060 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.495982 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 20:01:23.496060 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.496000 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2tgj9\"" Jul 07 20:01:23.496060 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.496016 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39-cni-binary-copy\") pod \"multus-additional-cni-plugins-vx82p\" (UID: \"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39\") " pod="openshift-multus/multus-additional-cni-plugins-vx82p" Jul 07 20:01:23.496203 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.496043 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-etc-kubernetes\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.496203 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.496122 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da37b4bb-9b02-4491-9d87-c4f56be7c7ad-metrics-certs\") pod \"network-metrics-daemon-qvph6\" (UID: \"da37b4bb-9b02-4491-9d87-c4f56be7c7ad\") " pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:01:23.496300 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.496207 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8nq5\" (UniqueName: \"kubernetes.io/projected/da37b4bb-9b02-4491-9d87-c4f56be7c7ad-kube-api-access-g8nq5\") pod \"network-metrics-daemon-qvph6\" (UID: \"da37b4bb-9b02-4491-9d87-c4f56be7c7ad\") " pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:01:23.496300 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.496242 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 07 20:01:23.496300 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.496261 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-prgz7\"" Jul 07 20:01:23.496300 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.496245 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2c31e770-b420-42cf-8bf6-a7f8540ed592-konnectivity-ca\") pod \"konnectivity-agent-jlj7d\" (UID: \"2c31e770-b420-42cf-8bf6-a7f8540ed592\") " pod="kube-system/konnectivity-agent-jlj7d" Jul 07 20:01:23.496507 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.496310 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3290b11-e223-477b-b593-745224204b1e-tmp-dir\") pod \"node-resolver-b8qtm\" (UID: \"f3290b11-e223-477b-b593-745224204b1e\") " pod="openshift-dns/node-resolver-b8qtm" Jul 07 20:01:23.496507 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.496470 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc753a68-481d-469b-88a3-37bbb48ac522-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dfbfw\" (UID: \"dc753a68-481d-469b-88a3-37bbb48ac522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfbfw" Jul 07 20:01:23.496611 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.496503 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc753a68-481d-469b-88a3-37bbb48ac522-socket-dir\") pod \"aws-ebs-csi-driver-node-dfbfw\" (UID: \"dc753a68-481d-469b-88a3-37bbb48ac522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfbfw" Jul 07 20:01:23.496611 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.496576 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgzjz\" (UniqueName: \"kubernetes.io/projected/dc753a68-481d-469b-88a3-37bbb48ac522-kube-api-access-bgzjz\") pod \"aws-ebs-csi-driver-node-dfbfw\" (UID: \"dc753a68-481d-469b-88a3-37bbb48ac522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfbfw" Jul 07 20:01:23.496706 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.496614 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljl7h\" (UniqueName: \"kubernetes.io/projected/f3290b11-e223-477b-b593-745224204b1e-kube-api-access-ljl7h\") pod \"node-resolver-b8qtm\" (UID: \"f3290b11-e223-477b-b593-745224204b1e\") " pod="openshift-dns/node-resolver-b8qtm" Jul 07 20:01:23.496706 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.496650 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghvws\" (UniqueName: \"kubernetes.io/projected/e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39-kube-api-access-ghvws\") pod \"multus-additional-cni-plugins-vx82p\" (UID: \"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39\") " pod="openshift-multus/multus-additional-cni-plugins-vx82p" Jul 07 20:01:23.496706 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.496679 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-etc-modprobe-d\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.496706 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.496699 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/19ef0176-73b8-465c-abda-754d10ddd21f-etc-tuned\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.496887 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.496765 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f3290b11-e223-477b-b593-745224204b1e-hosts-file\") pod \"node-resolver-b8qtm\" (UID: \"f3290b11-e223-477b-b593-745224204b1e\") " pod="openshift-dns/node-resolver-b8qtm" Jul 07 20:01:23.496887 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.496819 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-var-lib-kubelet\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.496887 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.496842 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19ef0176-73b8-465c-abda-754d10ddd21f-tmp\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.497031 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.496876 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spwbq\" (UniqueName: \"kubernetes.io/projected/92ea0d19-f83c-4b0d-969d-cf288f52c55a-kube-api-access-spwbq\") pod \"network-check-target-wj7w5\" (UID: \"92ea0d19-f83c-4b0d-969d-cf288f52c55a\") " pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:01:23.497031 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.496925 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39-system-cni-dir\") pod \"multus-additional-cni-plugins-vx82p\" (UID: \"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39\") " pod="openshift-multus/multus-additional-cni-plugins-vx82p" Jul 07 20:01:23.497031 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.496998 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39-cnibin\") pod \"multus-additional-cni-plugins-vx82p\" (UID: \"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39\") " pod="openshift-multus/multus-additional-cni-plugins-vx82p" Jul 07 20:01:23.497031 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.497017 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39-os-release\") pod \"multus-additional-cni-plugins-vx82p\" (UID: \"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39\") " pod="openshift-multus/multus-additional-cni-plugins-vx82p" Jul 07 20:01:23.497185 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.497046 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vx82p\" (UID: \"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39\") " pod="openshift-multus/multus-additional-cni-plugins-vx82p" Jul 07 20:01:23.497239 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.497165 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-run\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.497239 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.497209 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc753a68-481d-469b-88a3-37bbb48ac522-etc-selinux\") pod \"aws-ebs-csi-driver-node-dfbfw\" (UID: \"dc753a68-481d-469b-88a3-37bbb48ac522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfbfw" Jul 07 20:01:23.502361 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.501301 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 20:01:23.523474 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.523456 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-b2m5z" Jul 07 20:01:23.530680 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.530662 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-b2m5z" Jul 07 20:01:23.587160 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.587138 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 07 20:01:23.598273 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.598253 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-hostroot\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.598376 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.598279 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc753a68-481d-469b-88a3-37bbb48ac522-device-dir\") pod \"aws-ebs-csi-driver-node-dfbfw\" (UID: \"dc753a68-481d-469b-88a3-37bbb48ac522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfbfw" Jul 07 20:01:23.598376 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.598297 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da37b4bb-9b02-4491-9d87-c4f56be7c7ad-metrics-certs\") pod \"network-metrics-daemon-qvph6\" (UID: \"da37b4bb-9b02-4491-9d87-c4f56be7c7ad\") " pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:01:23.598376 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.598310 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2c31e770-b420-42cf-8bf6-a7f8540ed592-konnectivity-ca\") pod \"konnectivity-agent-jlj7d\" (UID: \"2c31e770-b420-42cf-8bf6-a7f8540ed592\") " pod="kube-system/konnectivity-agent-jlj7d" Jul 07 20:01:23.598376 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.598325 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-host-slash\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.598376 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.598341 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.598563 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.598364 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f34b812-e3ff-47b7-ad26-092401121393-ovn-node-metrics-cert\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.598563 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.598378 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc753a68-481d-469b-88a3-37bbb48ac522-device-dir\") pod \"aws-ebs-csi-driver-node-dfbfw\" (UID: \"dc753a68-481d-469b-88a3-37bbb48ac522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfbfw" Jul 07 20:01:23.598563 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.598419 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4q26\" (UniqueName: \"kubernetes.io/projected/f5a8c362-f805-466b-ae7b-ca28c4087e63-kube-api-access-v4q26\") pod \"node-ca-jbxhj\" (UID: \"f5a8c362-f805-466b-ae7b-ca28c4087e63\") " pod="openshift-image-registry/node-ca-jbxhj" Jul 07 20:01:23.598563 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.598485 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39-cni-binary-copy\") pod \"multus-additional-cni-plugins-vx82p\" (UID: \"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39\") " pod="openshift-multus/multus-additional-cni-plugins-vx82p" Jul 07 20:01:23.598563 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.598516 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-node-log\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.598563 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:23.598530 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 20:01:23.598563 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.598542 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-host-var-lib-cni-bin\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.598871 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.598570 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc753a68-481d-469b-88a3-37bbb48ac522-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dfbfw\" (UID: \"dc753a68-481d-469b-88a3-37bbb48ac522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfbfw" Jul 07 20:01:23.598871 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:23.598602 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da37b4bb-9b02-4491-9d87-c4f56be7c7ad-metrics-certs podName:da37b4bb-9b02-4491-9d87-c4f56be7c7ad nodeName:}" failed. No retries permitted until 2026-07-07 20:01:24.098581346 +0000 UTC m=+4.253043290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da37b4bb-9b02-4491-9d87-c4f56be7c7ad-metrics-certs") pod "network-metrics-daemon-qvph6" (UID: "da37b4bb-9b02-4491-9d87-c4f56be7c7ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 20:01:23.598871 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.598632 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc753a68-481d-469b-88a3-37bbb48ac522-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dfbfw\" (UID: \"dc753a68-481d-469b-88a3-37bbb48ac522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfbfw" Jul 07 20:01:23.598871 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.598650 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgzjz\" (UniqueName: \"kubernetes.io/projected/dc753a68-481d-469b-88a3-37bbb48ac522-kube-api-access-bgzjz\") pod \"aws-ebs-csi-driver-node-dfbfw\" (UID: \"dc753a68-481d-469b-88a3-37bbb48ac522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfbfw" Jul 07 20:01:23.598871 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.598685 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-cnibin\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.598871 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.598717 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-multus-socket-dir-parent\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.598871 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.598755 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghvws\" (UniqueName: \"kubernetes.io/projected/e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39-kube-api-access-ghvws\") pod \"multus-additional-cni-plugins-vx82p\" (UID: \"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39\") " pod="openshift-multus/multus-additional-cni-plugins-vx82p" Jul 07 20:01:23.598871 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.598780 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/19ef0176-73b8-465c-abda-754d10ddd21f-etc-tuned\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.598871 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.598812 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f3290b11-e223-477b-b593-745224204b1e-hosts-file\") pod \"node-resolver-b8qtm\" (UID: \"f3290b11-e223-477b-b593-745224204b1e\") " pod="openshift-dns/node-resolver-b8qtm" Jul 07 20:01:23.598871 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.598836 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-var-lib-openvswitch\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.598871 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.598860 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2c31e770-b420-42cf-8bf6-a7f8540ed592-konnectivity-ca\") pod \"konnectivity-agent-jlj7d\" (UID: \"2c31e770-b420-42cf-8bf6-a7f8540ed592\") " pod="kube-system/konnectivity-agent-jlj7d" Jul 07 20:01:23.599381 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.598861 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f5a8c362-f805-466b-ae7b-ca28c4087e63-serviceca\") pod \"node-ca-jbxhj\" (UID: \"f5a8c362-f805-466b-ae7b-ca28c4087e63\") " pod="openshift-image-registry/node-ca-jbxhj" Jul 07 20:01:23.599381 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.598916 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-host-var-lib-kubelet\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.599381 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.598932 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f3290b11-e223-477b-b593-745224204b1e-hosts-file\") pod \"node-resolver-b8qtm\" (UID: \"f3290b11-e223-477b-b593-745224204b1e\") " pod="openshift-dns/node-resolver-b8qtm" Jul 07 20:01:23.599381 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.598943 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-etc-kubernetes\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.599381 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.598968 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17142129-e574-4092-bd8d-0ffa3de6507a-kubelet-config\") pod \"global-pull-secret-syncer-zr7c2\" (UID: \"17142129-e574-4092-bd8d-0ffa3de6507a\") " pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:23.599381 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.598999 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-host-cni-netd\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.599381 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599019 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7f34b812-e3ff-47b7-ad26-092401121393-ovnkube-config\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.599381 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599034 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wcfs\" (UniqueName: \"kubernetes.io/projected/7f34b812-e3ff-47b7-ad26-092401121393-kube-api-access-2wcfs\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.599381 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599056 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39-cni-binary-copy\") pod \"multus-additional-cni-plugins-vx82p\" (UID: \"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39\") " pod="openshift-multus/multus-additional-cni-plugins-vx82p" Jul 07 20:01:23.599381 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599065 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-multus-cni-dir\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.599381 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599099 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vx82p\" (UID: \"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39\") " pod="openshift-multus/multus-additional-cni-plugins-vx82p" Jul 07 20:01:23.599381 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599121 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-run\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.599381 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599150 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc753a68-481d-469b-88a3-37bbb48ac522-etc-selinux\") pod \"aws-ebs-csi-driver-node-dfbfw\" (UID: \"dc753a68-481d-469b-88a3-37bbb48ac522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfbfw" Jul 07 20:01:23.599381 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599175 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-host-kubelet\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.599381 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599181 2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 07 20:01:23.599381 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599198 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-run-openvswitch\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.599381 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599221 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7f34b812-e3ff-47b7-ad26-092401121393-env-overrides\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.600141 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599196 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-run\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.600141 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599245 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5a8c362-f805-466b-ae7b-ca28c4087e63-host\") pod \"node-ca-jbxhj\" (UID: \"f5a8c362-f805-466b-ae7b-ca28c4087e63\") " pod="openshift-image-registry/node-ca-jbxhj" Jul 07 20:01:23.600141 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599269 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b90c45b-0580-4343-b306-5bcc726279a6-cni-binary-copy\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.600141 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599294 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc753a68-481d-469b-88a3-37bbb48ac522-etc-selinux\") pod \"aws-ebs-csi-driver-node-dfbfw\" (UID: \"dc753a68-481d-469b-88a3-37bbb48ac522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfbfw" Jul 07 20:01:23.600141 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599311 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-sys\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.600141 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599348 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-sys\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.600141 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599349 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-run-ovn\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.600141 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599376 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-host-run-ovn-kubernetes\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.600141 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599415 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-host-run-netns\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.600141 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599446 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-etc-sysctl-conf\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.600141 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599462 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-lib-modules\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.600141 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599477 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2c31e770-b420-42cf-8bf6-a7f8540ed592-agent-certs\") pod \"konnectivity-agent-jlj7d\" (UID: \"2c31e770-b420-42cf-8bf6-a7f8540ed592\") " pod="kube-system/konnectivity-agent-jlj7d" Jul 07 20:01:23.600141 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599510 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-os-release\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.600141 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599519 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vx82p\" (UID: \"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39\") " pod="openshift-multus/multus-additional-cni-plugins-vx82p" Jul 07 20:01:23.600141 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599545 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vx82p\" (UID: \"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39\") " pod="openshift-multus/multus-additional-cni-plugins-vx82p" Jul 07 20:01:23.600141 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599564 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-etc-sysconfig\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.600141 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599586 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-etc-sysctl-d\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.600907 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599601 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-etc-sysctl-conf\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.600907 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599608 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-etc-systemd\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.600907 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599611 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-lib-modules\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.600907 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599638 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwtp2\" (UniqueName: \"kubernetes.io/projected/19ef0176-73b8-465c-abda-754d10ddd21f-kube-api-access-nwtp2\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.600907 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599649 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-etc-systemd\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.600907 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599665 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc753a68-481d-469b-88a3-37bbb48ac522-registration-dir\") pod \"aws-ebs-csi-driver-node-dfbfw\" (UID: \"dc753a68-481d-469b-88a3-37bbb48ac522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfbfw" Jul 07 20:01:23.600907 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599692 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc753a68-481d-469b-88a3-37bbb48ac522-sys-fs\") pod \"aws-ebs-csi-driver-node-dfbfw\" (UID: \"dc753a68-481d-469b-88a3-37bbb48ac522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfbfw" Jul 07 20:01:23.600907 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599706 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-etc-sysconfig\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.600907 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599730 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8nq5\" (UniqueName: \"kubernetes.io/projected/da37b4bb-9b02-4491-9d87-c4f56be7c7ad-kube-api-access-g8nq5\") pod \"network-metrics-daemon-qvph6\" (UID: \"da37b4bb-9b02-4491-9d87-c4f56be7c7ad\") " pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:01:23.600907 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599753 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-etc-kubernetes\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.600907 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599773 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3290b11-e223-477b-b593-745224204b1e-tmp-dir\") pod \"node-resolver-b8qtm\" (UID: \"f3290b11-e223-477b-b593-745224204b1e\") " pod="openshift-dns/node-resolver-b8qtm" Jul 07 20:01:23.600907 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599792 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vx82p\" (UID: \"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39\") " pod="openshift-multus/multus-additional-cni-plugins-vx82p" Jul 07 20:01:23.600907 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599799 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-systemd-units\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.600907 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599806 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-etc-sysctl-d\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.600907 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599828 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc753a68-481d-469b-88a3-37bbb48ac522-registration-dir\") pod \"aws-ebs-csi-driver-node-dfbfw\" (UID: \"dc753a68-481d-469b-88a3-37bbb48ac522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfbfw" Jul 07 20:01:23.600907 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599830 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-system-cni-dir\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.600907 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599862 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc753a68-481d-469b-88a3-37bbb48ac522-sys-fs\") pod \"aws-ebs-csi-driver-node-dfbfw\" (UID: \"dc753a68-481d-469b-88a3-37bbb48ac522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfbfw" Jul 07 20:01:23.601420 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599877 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17142129-e574-4092-bd8d-0ffa3de6507a-dbus\") pod \"global-pull-secret-syncer-zr7c2\" (UID: \"17142129-e574-4092-bd8d-0ffa3de6507a\") " pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:23.601420 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599906 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc753a68-481d-469b-88a3-37bbb48ac522-socket-dir\") pod \"aws-ebs-csi-driver-node-dfbfw\" (UID: \"dc753a68-481d-469b-88a3-37bbb48ac522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfbfw" Jul 07 20:01:23.601420 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599931 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljl7h\" (UniqueName: \"kubernetes.io/projected/f3290b11-e223-477b-b593-745224204b1e-kube-api-access-ljl7h\") pod \"node-resolver-b8qtm\" (UID: \"f3290b11-e223-477b-b593-745224204b1e\") " pod="openshift-dns/node-resolver-b8qtm" Jul 07 20:01:23.601420 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599956 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-host-run-netns\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.601420 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.599982 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17142129-e574-4092-bd8d-0ffa3de6507a-original-pull-secret\") pod \"global-pull-secret-syncer-zr7c2\" (UID: \"17142129-e574-4092-bd8d-0ffa3de6507a\") " pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:23.601420 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600006 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-etc-modprobe-d\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.601420 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600032 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dc031eff-d6a1-44f6-803b-bc0fb0341bc0-iptables-alerter-script\") pod \"iptables-alerter-v6vf5\" (UID: \"dc031eff-d6a1-44f6-803b-bc0fb0341bc0\") " pod="openshift-network-operator/iptables-alerter-v6vf5" Jul 07 20:01:23.601420 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600032 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-etc-kubernetes\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.601420 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600067 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-etc-openvswitch\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.601420 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600111 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3290b11-e223-477b-b593-745224204b1e-tmp-dir\") pod \"node-resolver-b8qtm\" (UID: \"f3290b11-e223-477b-b593-745224204b1e\") " pod="openshift-dns/node-resolver-b8qtm" Jul 07 20:01:23.601420 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600108 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2l4z\" (UniqueName: \"kubernetes.io/projected/1b90c45b-0580-4343-b306-5bcc726279a6-kube-api-access-b2l4z\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.601420 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600150 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-var-lib-kubelet\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.601420 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600175 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19ef0176-73b8-465c-abda-754d10ddd21f-tmp\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.601420 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600202 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7f34b812-e3ff-47b7-ad26-092401121393-ovnkube-script-lib\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.601420 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600238 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-host-var-lib-cni-multus\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.601420 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600246 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-etc-modprobe-d\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.601420 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600273 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spwbq\" (UniqueName: \"kubernetes.io/projected/92ea0d19-f83c-4b0d-969d-cf288f52c55a-kube-api-access-spwbq\") pod \"network-check-target-wj7w5\" (UID: \"92ea0d19-f83c-4b0d-969d-cf288f52c55a\") " pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:01:23.601934 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600298 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-var-lib-kubelet\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.601934 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600300 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39-system-cni-dir\") pod \"multus-additional-cni-plugins-vx82p\" (UID: \"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39\") " pod="openshift-multus/multus-additional-cni-plugins-vx82p" Jul 07 20:01:23.601934 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600348 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39-cnibin\") pod \"multus-additional-cni-plugins-vx82p\" (UID: \"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39\") " pod="openshift-multus/multus-additional-cni-plugins-vx82p" Jul 07 20:01:23.601934 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600414 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39-system-cni-dir\") pod \"multus-additional-cni-plugins-vx82p\" (UID: \"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39\") " pod="openshift-multus/multus-additional-cni-plugins-vx82p" Jul 07 20:01:23.601934 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600454 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39-os-release\") pod \"multus-additional-cni-plugins-vx82p\" (UID: \"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39\") " pod="openshift-multus/multus-additional-cni-plugins-vx82p" Jul 07 20:01:23.601934 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600502 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc753a68-481d-469b-88a3-37bbb48ac522-socket-dir\") pod \"aws-ebs-csi-driver-node-dfbfw\" (UID: \"dc753a68-481d-469b-88a3-37bbb48ac522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfbfw" Jul 07 20:01:23.601934 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600576 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc031eff-d6a1-44f6-803b-bc0fb0341bc0-host-slash\") pod \"iptables-alerter-v6vf5\" (UID: \"dc031eff-d6a1-44f6-803b-bc0fb0341bc0\") " pod="openshift-network-operator/iptables-alerter-v6vf5" Jul 07 20:01:23.601934 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600602 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5pwp\" (UniqueName: \"kubernetes.io/projected/dc031eff-d6a1-44f6-803b-bc0fb0341bc0-kube-api-access-v5pwp\") pod \"iptables-alerter-v6vf5\" (UID: \"dc031eff-d6a1-44f6-803b-bc0fb0341bc0\") " pod="openshift-network-operator/iptables-alerter-v6vf5" Jul 07 20:01:23.601934 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600637 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-run-systemd\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.601934 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600651 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39-os-release\") pod \"multus-additional-cni-plugins-vx82p\" (UID: \"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39\") " pod="openshift-multus/multus-additional-cni-plugins-vx82p" Jul 07 20:01:23.601934 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600666 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-host-run-k8s-cni-cncf-io\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.601934 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600684 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39-cnibin\") pod \"multus-additional-cni-plugins-vx82p\" (UID: \"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39\") " pod="openshift-multus/multus-additional-cni-plugins-vx82p" Jul 07 20:01:23.601934 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600691 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-log-socket\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.601934 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600720 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-multus-conf-dir\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.601934 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600747 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1b90c45b-0580-4343-b306-5bcc726279a6-multus-daemon-config\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.601934 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600769 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-host-run-multus-certs\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.601934 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600807 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vx82p\" (UID: \"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39\") " pod="openshift-multus/multus-additional-cni-plugins-vx82p" Jul 07 20:01:23.602388 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600836 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-host\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.602388 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600861 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-host-cni-bin\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.602388 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.600915 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19ef0176-73b8-465c-abda-754d10ddd21f-host\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.602388 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.601201 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vx82p\" (UID: \"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39\") " pod="openshift-multus/multus-additional-cni-plugins-vx82p" Jul 07 20:01:23.603020 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.602994 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/19ef0176-73b8-465c-abda-754d10ddd21f-etc-tuned\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.603020 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.603010 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19ef0176-73b8-465c-abda-754d10ddd21f-tmp\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.603207 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.603192 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2c31e770-b420-42cf-8bf6-a7f8540ed592-agent-certs\") pod \"konnectivity-agent-jlj7d\" (UID: \"2c31e770-b420-42cf-8bf6-a7f8540ed592\") " pod="kube-system/konnectivity-agent-jlj7d" Jul 07 20:01:23.611958 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:23.611939 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 20:01:23.611958 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:23.611956 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 20:01:23.612091 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:23.611964 2565 projected.go:196] Error preparing data for projected volume kube-api-access-spwbq for pod openshift-network-diagnostics/network-check-target-wj7w5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 20:01:23.612091 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:23.612018 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92ea0d19-f83c-4b0d-969d-cf288f52c55a-kube-api-access-spwbq podName:92ea0d19-f83c-4b0d-969d-cf288f52c55a nodeName:}" failed. No retries permitted until 2026-07-07 20:01:24.1120036 +0000 UTC m=+4.266465547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-spwbq" (UniqueName: "kubernetes.io/projected/92ea0d19-f83c-4b0d-969d-cf288f52c55a-kube-api-access-spwbq") pod "network-check-target-wj7w5" (UID: "92ea0d19-f83c-4b0d-969d-cf288f52c55a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 20:01:23.612837 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.612813 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8nq5\" (UniqueName: \"kubernetes.io/projected/da37b4bb-9b02-4491-9d87-c4f56be7c7ad-kube-api-access-g8nq5\") pod \"network-metrics-daemon-qvph6\" (UID: \"da37b4bb-9b02-4491-9d87-c4f56be7c7ad\") " pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:01:23.614511 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.614481 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgzjz\" (UniqueName: \"kubernetes.io/projected/dc753a68-481d-469b-88a3-37bbb48ac522-kube-api-access-bgzjz\") pod \"aws-ebs-csi-driver-node-dfbfw\" (UID: \"dc753a68-481d-469b-88a3-37bbb48ac522\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfbfw" Jul 07 20:01:23.614846 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.614823 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwtp2\" (UniqueName: \"kubernetes.io/projected/19ef0176-73b8-465c-abda-754d10ddd21f-kube-api-access-nwtp2\") pod \"tuned-7gbnw\" (UID: \"19ef0176-73b8-465c-abda-754d10ddd21f\") " pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.615549 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.615526 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghvws\" (UniqueName: \"kubernetes.io/projected/e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39-kube-api-access-ghvws\") pod \"multus-additional-cni-plugins-vx82p\" (UID: \"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39\") " pod="openshift-multus/multus-additional-cni-plugins-vx82p" Jul 07 20:01:23.616002 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.615978 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljl7h\" (UniqueName: \"kubernetes.io/projected/f3290b11-e223-477b-b593-745224204b1e-kube-api-access-ljl7h\") pod \"node-resolver-b8qtm\" (UID: \"f3290b11-e223-477b-b593-745224204b1e\") " pod="openshift-dns/node-resolver-b8qtm" Jul 07 20:01:23.683441 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.683388 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-229.ec2.internal" event={"ID":"59bf4ed301a47b79107844633c8ed6ad","Type":"ContainerStarted","Data":"2ee03d5f441c0b4e8c2d01a947ede35539f0134c600f55a4e27d66cd747bd764"} Jul 07 20:01:23.684329 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.684310 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-229.ec2.internal" event={"ID":"da4ff7f5b4692b4311b38219eb74d174","Type":"ContainerStarted","Data":"ec9c5e0e4d0ea04ce2faa75ab7d747e184146c533717ad88fbe0a704a7074096"} Jul 07 20:01:23.701136 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701116 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-cnibin\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.701228 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701150 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-multus-socket-dir-parent\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.701228 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701174 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-var-lib-openvswitch\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.701228 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701193 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f5a8c362-f805-466b-ae7b-ca28c4087e63-serviceca\") pod \"node-ca-jbxhj\" (UID: \"f5a8c362-f805-466b-ae7b-ca28c4087e63\") " pod="openshift-image-registry/node-ca-jbxhj" Jul 07 20:01:23.701228 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701207 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-cnibin\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.701228 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701218 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-host-var-lib-kubelet\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.701503 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701240 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-etc-kubernetes\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.701503 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701250 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-multus-socket-dir-parent\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.701503 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701263 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17142129-e574-4092-bd8d-0ffa3de6507a-kubelet-config\") pod \"global-pull-secret-syncer-zr7c2\" (UID: \"17142129-e574-4092-bd8d-0ffa3de6507a\") " pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:23.701503 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701268 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-var-lib-openvswitch\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.701503 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701285 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-host-var-lib-kubelet\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.701503 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701287 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-host-cni-netd\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.701503 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701304 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-etc-kubernetes\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.701503 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701326 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7f34b812-e3ff-47b7-ad26-092401121393-ovnkube-config\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.701503 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701335 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17142129-e574-4092-bd8d-0ffa3de6507a-kubelet-config\") pod \"global-pull-secret-syncer-zr7c2\" (UID: \"17142129-e574-4092-bd8d-0ffa3de6507a\") " pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:23.701503 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701328 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-host-cni-netd\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.701503 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701352 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wcfs\" (UniqueName: \"kubernetes.io/projected/7f34b812-e3ff-47b7-ad26-092401121393-kube-api-access-2wcfs\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.701503 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701434 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-multus-cni-dir\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.701503 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701465 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-host-kubelet\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.701503 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701490 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-run-openvswitch\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.702154 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701512 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7f34b812-e3ff-47b7-ad26-092401121393-env-overrides\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.702154 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701535 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5a8c362-f805-466b-ae7b-ca28c4087e63-host\") pod \"node-ca-jbxhj\" (UID: \"f5a8c362-f805-466b-ae7b-ca28c4087e63\") " pod="openshift-image-registry/node-ca-jbxhj" Jul 07 20:01:23.702154 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701561 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b90c45b-0580-4343-b306-5bcc726279a6-cni-binary-copy\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.702154 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701585 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-run-ovn\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.702154 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701608 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-host-run-ovn-kubernetes\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.702154 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701610 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-multus-cni-dir\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.702154 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701654 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f5a8c362-f805-466b-ae7b-ca28c4087e63-host\") pod \"node-ca-jbxhj\" (UID: \"f5a8c362-f805-466b-ae7b-ca28c4087e63\") " pod="openshift-image-registry/node-ca-jbxhj" Jul 07 20:01:23.702154 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701676 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-host-run-netns\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.702154 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701694 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-run-openvswitch\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.702154 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701707 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-os-release\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.702154 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701737 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-host-kubelet\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.702154 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701742 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-systemd-units\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.702154 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701764 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-system-cni-dir\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.702154 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701656 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-host-run-ovn-kubernetes\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.702154 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701656 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f5a8c362-f805-466b-ae7b-ca28c4087e63-serviceca\") pod \"node-ca-jbxhj\" (UID: \"f5a8c362-f805-466b-ae7b-ca28c4087e63\") " pod="openshift-image-registry/node-ca-jbxhj" Jul 07 20:01:23.702154 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701776 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-run-ovn\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.702154 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701785 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17142129-e574-4092-bd8d-0ffa3de6507a-dbus\") pod \"global-pull-secret-syncer-zr7c2\" (UID: \"17142129-e574-4092-bd8d-0ffa3de6507a\") " pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:23.702154 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701818 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-systemd-units\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.702989 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701840 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-os-release\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.702989 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701840 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17142129-e574-4092-bd8d-0ffa3de6507a-dbus\") pod \"global-pull-secret-syncer-zr7c2\" (UID: \"17142129-e574-4092-bd8d-0ffa3de6507a\") " pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:23.702989 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701854 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-host-run-netns\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.702989 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701887 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-host-run-netns\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.702989 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701894 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-system-cni-dir\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.702989 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701892 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17142129-e574-4092-bd8d-0ffa3de6507a-original-pull-secret\") pod \"global-pull-secret-syncer-zr7c2\" (UID: \"17142129-e574-4092-bd8d-0ffa3de6507a\") " pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:23.702989 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701931 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dc031eff-d6a1-44f6-803b-bc0fb0341bc0-iptables-alerter-script\") pod \"iptables-alerter-v6vf5\" (UID: \"dc031eff-d6a1-44f6-803b-bc0fb0341bc0\") " pod="openshift-network-operator/iptables-alerter-v6vf5" Jul 07 20:01:23.702989 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701938 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-host-run-netns\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.702989 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701940 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7f34b812-e3ff-47b7-ad26-092401121393-env-overrides\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.702989 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701947 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-etc-openvswitch\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.702989 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701966 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-etc-openvswitch\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.702989 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:23.701961 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 20:01:23.702989 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.701987 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2l4z\" (UniqueName: \"kubernetes.io/projected/1b90c45b-0580-4343-b306-5bcc726279a6-kube-api-access-b2l4z\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.702989 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702015 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7f34b812-e3ff-47b7-ad26-092401121393-ovnkube-script-lib\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.702989 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:23.702023 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17142129-e574-4092-bd8d-0ffa3de6507a-original-pull-secret podName:17142129-e574-4092-bd8d-0ffa3de6507a nodeName:}" failed. No retries permitted until 2026-07-07 20:01:24.20201002 +0000 UTC m=+4.356471950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17142129-e574-4092-bd8d-0ffa3de6507a-original-pull-secret") pod "global-pull-secret-syncer-zr7c2" (UID: "17142129-e574-4092-bd8d-0ffa3de6507a") : object "kube-system"/"original-pull-secret" not registered Jul 07 20:01:23.702989 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702044 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-host-var-lib-cni-multus\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.702989 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702079 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc031eff-d6a1-44f6-803b-bc0fb0341bc0-host-slash\") pod \"iptables-alerter-v6vf5\" (UID: \"dc031eff-d6a1-44f6-803b-bc0fb0341bc0\") " pod="openshift-network-operator/iptables-alerter-v6vf5" Jul 07 20:01:23.703662 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702102 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5pwp\" (UniqueName: \"kubernetes.io/projected/dc031eff-d6a1-44f6-803b-bc0fb0341bc0-kube-api-access-v5pwp\") pod \"iptables-alerter-v6vf5\" (UID: \"dc031eff-d6a1-44f6-803b-bc0fb0341bc0\") " pod="openshift-network-operator/iptables-alerter-v6vf5" Jul 07 20:01:23.703662 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702124 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-run-systemd\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.703662 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702146 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-host-run-k8s-cni-cncf-io\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.703662 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702168 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-log-socket\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.703662 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702188 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b90c45b-0580-4343-b306-5bcc726279a6-cni-binary-copy\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.703662 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702192 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-multus-conf-dir\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.703662 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702222 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-multus-conf-dir\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.703662 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702238 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1b90c45b-0580-4343-b306-5bcc726279a6-multus-daemon-config\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.703662 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702261 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-run-systemd\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.703662 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702264 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-host-run-multus-certs\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.703662 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702293 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-host-run-k8s-cni-cncf-io\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.703662 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702325 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-host-cni-bin\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.703662 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702331 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-log-socket\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.703662 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702370 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-host-var-lib-cni-multus\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.703662 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702295 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-host-cni-bin\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.703662 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702366 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-host-run-multus-certs\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.703662 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702383 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc031eff-d6a1-44f6-803b-bc0fb0341bc0-host-slash\") pod \"iptables-alerter-v6vf5\" (UID: \"dc031eff-d6a1-44f6-803b-bc0fb0341bc0\") " pod="openshift-network-operator/iptables-alerter-v6vf5" Jul 07 20:01:23.703662 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702427 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-hostroot\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.704309 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702459 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-host-slash\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.704309 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702497 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-hostroot\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.704309 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702479 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dc031eff-d6a1-44f6-803b-bc0fb0341bc0-iptables-alerter-script\") pod \"iptables-alerter-v6vf5\" (UID: \"dc031eff-d6a1-44f6-803b-bc0fb0341bc0\") " pod="openshift-network-operator/iptables-alerter-v6vf5" Jul 07 20:01:23.704309 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702520 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-host-slash\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.704309 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702514 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7f34b812-e3ff-47b7-ad26-092401121393-ovnkube-script-lib\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.704309 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702528 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.704309 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702560 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.704309 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702564 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7f34b812-e3ff-47b7-ad26-092401121393-ovnkube-config\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.704309 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702570 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f34b812-e3ff-47b7-ad26-092401121393-ovn-node-metrics-cert\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.704309 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702595 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4q26\" (UniqueName: \"kubernetes.io/projected/f5a8c362-f805-466b-ae7b-ca28c4087e63-kube-api-access-v4q26\") pod \"node-ca-jbxhj\" (UID: \"f5a8c362-f805-466b-ae7b-ca28c4087e63\") " pod="openshift-image-registry/node-ca-jbxhj" Jul 07 20:01:23.704309 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702619 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-node-log\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.704309 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702641 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-host-var-lib-cni-bin\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.704309 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702686 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7f34b812-e3ff-47b7-ad26-092401121393-node-log\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.704309 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702696 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b90c45b-0580-4343-b306-5bcc726279a6-host-var-lib-cni-bin\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.704309 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.702716 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1b90c45b-0580-4343-b306-5bcc726279a6-multus-daemon-config\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.704874 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.704490 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f34b812-e3ff-47b7-ad26-092401121393-ovn-node-metrics-cert\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.713973 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.713950 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2l4z\" (UniqueName: \"kubernetes.io/projected/1b90c45b-0580-4343-b306-5bcc726279a6-kube-api-access-b2l4z\") pod \"multus-25ms9\" (UID: \"1b90c45b-0580-4343-b306-5bcc726279a6\") " pod="openshift-multus/multus-25ms9" Jul 07 20:01:23.714325 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.714306 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wcfs\" (UniqueName: \"kubernetes.io/projected/7f34b812-e3ff-47b7-ad26-092401121393-kube-api-access-2wcfs\") pod \"ovnkube-node-6skbt\" (UID: \"7f34b812-e3ff-47b7-ad26-092401121393\") " pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.714435 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.714424 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4q26\" (UniqueName: \"kubernetes.io/projected/f5a8c362-f805-466b-ae7b-ca28c4087e63-kube-api-access-v4q26\") pod \"node-ca-jbxhj\" (UID: \"f5a8c362-f805-466b-ae7b-ca28c4087e63\") " pod="openshift-image-registry/node-ca-jbxhj" Jul 07 20:01:23.714534 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.714511 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5pwp\" (UniqueName: \"kubernetes.io/projected/dc031eff-d6a1-44f6-803b-bc0fb0341bc0-kube-api-access-v5pwp\") pod \"iptables-alerter-v6vf5\" (UID: \"dc031eff-d6a1-44f6-803b-bc0fb0341bc0\") " pod="openshift-network-operator/iptables-alerter-v6vf5" Jul 07 20:01:23.787821 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.787751 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfbfw" Jul 07 20:01:23.795332 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.795307 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vx82p" Jul 07 20:01:23.806172 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.806150 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" Jul 07 20:01:23.810902 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.810884 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jlj7d" Jul 07 20:01:23.817870 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.817854 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b8qtm" Jul 07 20:01:23.825623 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.825607 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v6vf5" Jul 07 20:01:23.832487 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.832470 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:23.839448 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.839428 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jbxhj" Jul 07 20:01:23.844924 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:23.844905 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-25ms9" Jul 07 20:01:24.079808 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:24.079771 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ef0176_73b8_465c_abda_754d10ddd21f.slice/crio-cb06eab7b4d1580e4ce3a436efb93ca6cee94b373b7f800f4f4cdc970ce621f5 WatchSource:0}: Error finding container cb06eab7b4d1580e4ce3a436efb93ca6cee94b373b7f800f4f4cdc970ce621f5: Status 404 returned error can't find the container with id cb06eab7b4d1580e4ce3a436efb93ca6cee94b373b7f800f4f4cdc970ce621f5 Jul 07 20:01:24.080331 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:24.080306 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5a8c362_f805_466b_ae7b_ca28c4087e63.slice/crio-24c425b09ba197e23102b368878b5530bf2ee8e351f4efb3a5de1f27c6c40bec WatchSource:0}: Error finding container 24c425b09ba197e23102b368878b5530bf2ee8e351f4efb3a5de1f27c6c40bec: Status 404 returned error can't find the container with id 24c425b09ba197e23102b368878b5530bf2ee8e351f4efb3a5de1f27c6c40bec Jul 07 20:01:24.086454 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:24.086429 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3290b11_e223_477b_b593_745224204b1e.slice/crio-5d77e41d0a03750888094475b4f223d716b3de5c2cb38527e675bc67fb27f67b WatchSource:0}: Error finding container 5d77e41d0a03750888094475b4f223d716b3de5c2cb38527e675bc67fb27f67b: Status 404 returned error can't find the container with id 5d77e41d0a03750888094475b4f223d716b3de5c2cb38527e675bc67fb27f67b Jul 07 20:01:24.088292 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:24.088272 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b90c45b_0580_4343_b306_5bcc726279a6.slice/crio-f0cc2e51eec2cf22d866a2170e415ec801f3a1237a59faada7c29b1011ec5607 WatchSource:0}: Error finding container f0cc2e51eec2cf22d866a2170e415ec801f3a1237a59faada7c29b1011ec5607: Status 404 returned error can't find the container with id f0cc2e51eec2cf22d866a2170e415ec801f3a1237a59faada7c29b1011ec5607 Jul 07 20:01:24.089159 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:24.089134 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc753a68_481d_469b_88a3_37bbb48ac522.slice/crio-9b4b39cb595fc48fe8bb7f47cd7f7de3f1fb447c1f417e60603c3d9740b44598 WatchSource:0}: Error finding container 9b4b39cb595fc48fe8bb7f47cd7f7de3f1fb447c1f417e60603c3d9740b44598: Status 404 returned error can't find the container with id 9b4b39cb595fc48fe8bb7f47cd7f7de3f1fb447c1f417e60603c3d9740b44598 Jul 07 20:01:24.090121 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:24.090094 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1065bc7_1b4f_4a1f_80e8_6c75f45f9b39.slice/crio-a8fb609beafecd2523986e0956f3f2bb6c3c4266f4051e1dc12f1ff05daa9e1a WatchSource:0}: Error finding container a8fb609beafecd2523986e0956f3f2bb6c3c4266f4051e1dc12f1ff05daa9e1a: Status 404 returned error can't find the container with id a8fb609beafecd2523986e0956f3f2bb6c3c4266f4051e1dc12f1ff05daa9e1a Jul 07 20:01:24.090713 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:24.090664 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c31e770_b420_42cf_8bf6_a7f8540ed592.slice/crio-05f5b2d7131a36155c4dfd8931dadb1118a618ab8204ddecabeab1c832717333 WatchSource:0}: Error finding container 05f5b2d7131a36155c4dfd8931dadb1118a618ab8204ddecabeab1c832717333: Status 404 returned error can't find the container with id 05f5b2d7131a36155c4dfd8931dadb1118a618ab8204ddecabeab1c832717333 Jul 07 20:01:24.091549 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:24.091526 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc031eff_d6a1_44f6_803b_bc0fb0341bc0.slice/crio-23b7fd8032df6ce2a2d4d25643a3b577361f7e941adf2208417d0039af0f052f WatchSource:0}: Error finding container 23b7fd8032df6ce2a2d4d25643a3b577361f7e941adf2208417d0039af0f052f: Status 404 returned error can't find the container with id 23b7fd8032df6ce2a2d4d25643a3b577361f7e941adf2208417d0039af0f052f Jul 07 20:01:24.092789 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:01:24.092775 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f34b812_e3ff_47b7_ad26_092401121393.slice/crio-0af61ab0e3f15ad34bc9eb1e7c8f0e3135a42a70d9d66cd941eaffe87a3ab0ee WatchSource:0}: Error finding container 0af61ab0e3f15ad34bc9eb1e7c8f0e3135a42a70d9d66cd941eaffe87a3ab0ee: Status 404 returned error can't find the container with id 0af61ab0e3f15ad34bc9eb1e7c8f0e3135a42a70d9d66cd941eaffe87a3ab0ee Jul 07 20:01:24.105621 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:24.105603 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da37b4bb-9b02-4491-9d87-c4f56be7c7ad-metrics-certs\") pod \"network-metrics-daemon-qvph6\" (UID: \"da37b4bb-9b02-4491-9d87-c4f56be7c7ad\") " pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:01:24.105719 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:24.105704 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 20:01:24.105787 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:24.105750 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da37b4bb-9b02-4491-9d87-c4f56be7c7ad-metrics-certs podName:da37b4bb-9b02-4491-9d87-c4f56be7c7ad nodeName:}" failed. No retries permitted until 2026-07-07 20:01:25.105734368 +0000 UTC m=+5.260196314 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da37b4bb-9b02-4491-9d87-c4f56be7c7ad-metrics-certs") pod "network-metrics-daemon-qvph6" (UID: "da37b4bb-9b02-4491-9d87-c4f56be7c7ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 20:01:24.206212 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:24.206189 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spwbq\" (UniqueName: \"kubernetes.io/projected/92ea0d19-f83c-4b0d-969d-cf288f52c55a-kube-api-access-spwbq\") pod \"network-check-target-wj7w5\" (UID: \"92ea0d19-f83c-4b0d-969d-cf288f52c55a\") " pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:01:24.206317 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:24.206244 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17142129-e574-4092-bd8d-0ffa3de6507a-original-pull-secret\") pod \"global-pull-secret-syncer-zr7c2\" (UID: \"17142129-e574-4092-bd8d-0ffa3de6507a\") " pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:24.206363 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:24.206324 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 20:01:24.206363 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:24.206339 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 20:01:24.206363 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:24.206356 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 20:01:24.206472 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:24.206366 2565 projected.go:196] Error preparing data for projected volume kube-api-access-spwbq for pod openshift-network-diagnostics/network-check-target-wj7w5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 20:01:24.206472 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:24.206370 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17142129-e574-4092-bd8d-0ffa3de6507a-original-pull-secret podName:17142129-e574-4092-bd8d-0ffa3de6507a nodeName:}" failed. No retries permitted until 2026-07-07 20:01:25.206358079 +0000 UTC m=+5.360820009 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17142129-e574-4092-bd8d-0ffa3de6507a-original-pull-secret") pod "global-pull-secret-syncer-zr7c2" (UID: "17142129-e574-4092-bd8d-0ffa3de6507a") : object "kube-system"/"original-pull-secret" not registered Jul 07 20:01:24.206472 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:24.206427 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92ea0d19-f83c-4b0d-969d-cf288f52c55a-kube-api-access-spwbq podName:92ea0d19-f83c-4b0d-969d-cf288f52c55a nodeName:}" failed. No retries permitted until 2026-07-07 20:01:25.206392878 +0000 UTC m=+5.360854805 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-spwbq" (UniqueName: "kubernetes.io/projected/92ea0d19-f83c-4b0d-969d-cf288f52c55a-kube-api-access-spwbq") pod "network-check-target-wj7w5" (UID: "92ea0d19-f83c-4b0d-969d-cf288f52c55a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 20:01:24.533104 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:24.532871 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 19:56:23 +0000 UTC" deadline="2028-03-12 18:10:10.037865524 +0000 UTC" Jul 07 20:01:24.533104 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:24.533101 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14734h8m45.504771186s" Jul 07 20:01:24.687899 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:24.687861 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vx82p" event={"ID":"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39","Type":"ContainerStarted","Data":"a8fb609beafecd2523986e0956f3f2bb6c3c4266f4051e1dc12f1ff05daa9e1a"} Jul 07 20:01:24.689316 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:24.689286 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b8qtm" event={"ID":"f3290b11-e223-477b-b593-745224204b1e","Type":"ContainerStarted","Data":"5d77e41d0a03750888094475b4f223d716b3de5c2cb38527e675bc67fb27f67b"} Jul 07 20:01:24.691391 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:24.691362 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jbxhj" event={"ID":"f5a8c362-f805-466b-ae7b-ca28c4087e63","Type":"ContainerStarted","Data":"24c425b09ba197e23102b368878b5530bf2ee8e351f4efb3a5de1f27c6c40bec"} Jul 07 20:01:24.692705 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:24.692684 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" event={"ID":"19ef0176-73b8-465c-abda-754d10ddd21f","Type":"ContainerStarted","Data":"cb06eab7b4d1580e4ce3a436efb93ca6cee94b373b7f800f4f4cdc970ce621f5"} Jul 07 20:01:24.694017 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:24.693992 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v6vf5" event={"ID":"dc031eff-d6a1-44f6-803b-bc0fb0341bc0","Type":"ContainerStarted","Data":"23b7fd8032df6ce2a2d4d25643a3b577361f7e941adf2208417d0039af0f052f"} Jul 07 20:01:24.695217 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:24.695190 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-25ms9" event={"ID":"1b90c45b-0580-4343-b306-5bcc726279a6","Type":"ContainerStarted","Data":"f0cc2e51eec2cf22d866a2170e415ec801f3a1237a59faada7c29b1011ec5607"} Jul 07 20:01:24.696982 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:24.696957 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfbfw" event={"ID":"dc753a68-481d-469b-88a3-37bbb48ac522","Type":"ContainerStarted","Data":"9b4b39cb595fc48fe8bb7f47cd7f7de3f1fb447c1f417e60603c3d9740b44598"} Jul 07 20:01:24.702296 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:24.700055 2565 generic.go:356] "Generic (PLEG): container finished" podID="da4ff7f5b4692b4311b38219eb74d174" containerID="b493384d2ddbc80fb5150127dba7116565bc87f044224b22ca8049a632c9921d" exitCode=0 Jul 07 20:01:24.702296 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:24.700212 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-229.ec2.internal" event={"ID":"da4ff7f5b4692b4311b38219eb74d174","Type":"ContainerDied","Data":"b493384d2ddbc80fb5150127dba7116565bc87f044224b22ca8049a632c9921d"} Jul 07 20:01:24.703938 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:24.703913 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" event={"ID":"7f34b812-e3ff-47b7-ad26-092401121393","Type":"ContainerStarted","Data":"0af61ab0e3f15ad34bc9eb1e7c8f0e3135a42a70d9d66cd941eaffe87a3ab0ee"} Jul 07 20:01:24.705306 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:24.705280 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jlj7d" event={"ID":"2c31e770-b420-42cf-8bf6-a7f8540ed592","Type":"ContainerStarted","Data":"05f5b2d7131a36155c4dfd8931dadb1118a618ab8204ddecabeab1c832717333"} Jul 07 20:01:24.822213 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:24.821760 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 20:01:25.113094 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:25.113012 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da37b4bb-9b02-4491-9d87-c4f56be7c7ad-metrics-certs\") pod \"network-metrics-daemon-qvph6\" (UID: \"da37b4bb-9b02-4491-9d87-c4f56be7c7ad\") " pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:01:25.113306 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:25.113191 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 20:01:25.113306 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:25.113257 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da37b4bb-9b02-4491-9d87-c4f56be7c7ad-metrics-certs podName:da37b4bb-9b02-4491-9d87-c4f56be7c7ad nodeName:}" failed. No retries permitted until 2026-07-07 20:01:27.113237889 +0000 UTC m=+7.267699831 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da37b4bb-9b02-4491-9d87-c4f56be7c7ad-metrics-certs") pod "network-metrics-daemon-qvph6" (UID: "da37b4bb-9b02-4491-9d87-c4f56be7c7ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 20:01:25.213783 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:25.213748 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17142129-e574-4092-bd8d-0ffa3de6507a-original-pull-secret\") pod \"global-pull-secret-syncer-zr7c2\" (UID: \"17142129-e574-4092-bd8d-0ffa3de6507a\") " pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:25.213964 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:25.213800 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spwbq\" (UniqueName: \"kubernetes.io/projected/92ea0d19-f83c-4b0d-969d-cf288f52c55a-kube-api-access-spwbq\") pod \"network-check-target-wj7w5\" (UID: \"92ea0d19-f83c-4b0d-969d-cf288f52c55a\") " pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:01:25.213964 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:25.213939 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 20:01:25.213964 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:25.213960 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 20:01:25.214127 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:25.213973 2565 projected.go:196] Error preparing data for projected volume kube-api-access-spwbq for pod openshift-network-diagnostics/network-check-target-wj7w5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 20:01:25.214127 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:25.214031 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92ea0d19-f83c-4b0d-969d-cf288f52c55a-kube-api-access-spwbq podName:92ea0d19-f83c-4b0d-969d-cf288f52c55a nodeName:}" failed. No retries permitted until 2026-07-07 20:01:27.214013548 +0000 UTC m=+7.368475478 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-spwbq" (UniqueName: "kubernetes.io/projected/92ea0d19-f83c-4b0d-969d-cf288f52c55a-kube-api-access-spwbq") pod "network-check-target-wj7w5" (UID: "92ea0d19-f83c-4b0d-969d-cf288f52c55a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 20:01:25.214456 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:25.214422 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 20:01:25.214544 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:25.214474 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17142129-e574-4092-bd8d-0ffa3de6507a-original-pull-secret podName:17142129-e574-4092-bd8d-0ffa3de6507a nodeName:}" failed. No retries permitted until 2026-07-07 20:01:27.214458621 +0000 UTC m=+7.368920554 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17142129-e574-4092-bd8d-0ffa3de6507a-original-pull-secret") pod "global-pull-secret-syncer-zr7c2" (UID: "17142129-e574-4092-bd8d-0ffa3de6507a") : object "kube-system"/"original-pull-secret" not registered Jul 07 20:01:25.533924 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:25.533867 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 19:56:23 +0000 UTC" deadline="2027-12-08 14:47:45.878722731 +0000 UTC" Jul 07 20:01:25.533924 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:25.533922 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12450h46m20.344804381s" Jul 07 20:01:25.681285 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:25.681252 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:25.681474 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:25.681375 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr7c2" podUID="17142129-e574-4092-bd8d-0ffa3de6507a" Jul 07 20:01:25.681474 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:25.681471 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:01:25.681603 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:25.681558 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvph6" podUID="da37b4bb-9b02-4491-9d87-c4f56be7c7ad" Jul 07 20:01:25.681657 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:25.681620 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:01:25.681705 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:25.681687 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wj7w5" podUID="92ea0d19-f83c-4b0d-969d-cf288f52c55a" Jul 07 20:01:25.726061 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:25.725828 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-229.ec2.internal" event={"ID":"da4ff7f5b4692b4311b38219eb74d174","Type":"ContainerStarted","Data":"24832da92d92c78b8fd66ac598119090acd421e19827de66a04454d9db779955"} Jul 07 20:01:27.129825 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:27.129792 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da37b4bb-9b02-4491-9d87-c4f56be7c7ad-metrics-certs\") pod \"network-metrics-daemon-qvph6\" (UID: \"da37b4bb-9b02-4491-9d87-c4f56be7c7ad\") " pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:01:27.130302 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:27.130000 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 20:01:27.130302 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:27.130062 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da37b4bb-9b02-4491-9d87-c4f56be7c7ad-metrics-certs podName:da37b4bb-9b02-4491-9d87-c4f56be7c7ad nodeName:}" failed. No retries permitted until 2026-07-07 20:01:31.130043002 +0000 UTC m=+11.284504932 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da37b4bb-9b02-4491-9d87-c4f56be7c7ad-metrics-certs") pod "network-metrics-daemon-qvph6" (UID: "da37b4bb-9b02-4491-9d87-c4f56be7c7ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 20:01:27.231117 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:27.231084 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17142129-e574-4092-bd8d-0ffa3de6507a-original-pull-secret\") pod \"global-pull-secret-syncer-zr7c2\" (UID: \"17142129-e574-4092-bd8d-0ffa3de6507a\") " pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:27.231309 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:27.231141 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spwbq\" (UniqueName: \"kubernetes.io/projected/92ea0d19-f83c-4b0d-969d-cf288f52c55a-kube-api-access-spwbq\") pod \"network-check-target-wj7w5\" (UID: \"92ea0d19-f83c-4b0d-969d-cf288f52c55a\") " pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:01:27.231309 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:27.231269 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 20:01:27.231309 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:27.231292 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 20:01:27.231309 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:27.231306 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 20:01:27.231546 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:27.231318 2565 projected.go:196] Error preparing data for projected volume kube-api-access-spwbq for pod openshift-network-diagnostics/network-check-target-wj7w5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 20:01:27.231546 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:27.231349 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17142129-e574-4092-bd8d-0ffa3de6507a-original-pull-secret podName:17142129-e574-4092-bd8d-0ffa3de6507a nodeName:}" failed. No retries permitted until 2026-07-07 20:01:31.231328617 +0000 UTC m=+11.385790556 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17142129-e574-4092-bd8d-0ffa3de6507a-original-pull-secret") pod "global-pull-secret-syncer-zr7c2" (UID: "17142129-e574-4092-bd8d-0ffa3de6507a") : object "kube-system"/"original-pull-secret" not registered Jul 07 20:01:27.231546 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:27.231373 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92ea0d19-f83c-4b0d-969d-cf288f52c55a-kube-api-access-spwbq podName:92ea0d19-f83c-4b0d-969d-cf288f52c55a nodeName:}" failed. No retries permitted until 2026-07-07 20:01:31.231356518 +0000 UTC m=+11.385818465 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-spwbq" (UniqueName: "kubernetes.io/projected/92ea0d19-f83c-4b0d-969d-cf288f52c55a-kube-api-access-spwbq") pod "network-check-target-wj7w5" (UID: "92ea0d19-f83c-4b0d-969d-cf288f52c55a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 20:01:27.682932 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:27.682466 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:27.682932 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:27.682486 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:01:27.682932 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:27.682486 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:01:27.682932 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:27.682583 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr7c2" podUID="17142129-e574-4092-bd8d-0ffa3de6507a" Jul 07 20:01:27.682932 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:27.682721 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvph6" podUID="da37b4bb-9b02-4491-9d87-c4f56be7c7ad" Jul 07 20:01:27.682932 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:27.682818 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wj7w5" podUID="92ea0d19-f83c-4b0d-969d-cf288f52c55a" Jul 07 20:01:29.679123 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:29.679079 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:01:29.679123 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:29.679122 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:29.679645 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:29.679081 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:01:29.679645 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:29.679218 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wj7w5" podUID="92ea0d19-f83c-4b0d-969d-cf288f52c55a" Jul 07 20:01:29.679645 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:29.679417 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvph6" podUID="da37b4bb-9b02-4491-9d87-c4f56be7c7ad" Jul 07 20:01:29.679645 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:29.679507 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr7c2" podUID="17142129-e574-4092-bd8d-0ffa3de6507a" Jul 07 20:01:31.162223 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:31.162170 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da37b4bb-9b02-4491-9d87-c4f56be7c7ad-metrics-certs\") pod \"network-metrics-daemon-qvph6\" (UID: \"da37b4bb-9b02-4491-9d87-c4f56be7c7ad\") " pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:01:31.162701 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:31.162328 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 20:01:31.162701 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:31.162418 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da37b4bb-9b02-4491-9d87-c4f56be7c7ad-metrics-certs podName:da37b4bb-9b02-4491-9d87-c4f56be7c7ad nodeName:}" failed. No retries permitted until 2026-07-07 20:01:39.162379431 +0000 UTC m=+19.316841378 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da37b4bb-9b02-4491-9d87-c4f56be7c7ad-metrics-certs") pod "network-metrics-daemon-qvph6" (UID: "da37b4bb-9b02-4491-9d87-c4f56be7c7ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 20:01:31.262764 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:31.262731 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17142129-e574-4092-bd8d-0ffa3de6507a-original-pull-secret\") pod \"global-pull-secret-syncer-zr7c2\" (UID: \"17142129-e574-4092-bd8d-0ffa3de6507a\") " pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:31.262876 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:31.262782 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spwbq\" (UniqueName: \"kubernetes.io/projected/92ea0d19-f83c-4b0d-969d-cf288f52c55a-kube-api-access-spwbq\") pod \"network-check-target-wj7w5\" (UID: \"92ea0d19-f83c-4b0d-969d-cf288f52c55a\") " pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:01:31.262925 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:31.262878 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 20:01:31.262982 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:31.262942 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 20:01:31.262982 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:31.262964 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 20:01:31.262982 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:31.262977 2565 projected.go:196] Error preparing data for projected volume kube-api-access-spwbq for pod openshift-network-diagnostics/network-check-target-wj7w5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 20:01:31.263131 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:31.262944 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17142129-e574-4092-bd8d-0ffa3de6507a-original-pull-secret podName:17142129-e574-4092-bd8d-0ffa3de6507a nodeName:}" failed. No retries permitted until 2026-07-07 20:01:39.262925937 +0000 UTC m=+19.417387864 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17142129-e574-4092-bd8d-0ffa3de6507a-original-pull-secret") pod "global-pull-secret-syncer-zr7c2" (UID: "17142129-e574-4092-bd8d-0ffa3de6507a") : object "kube-system"/"original-pull-secret" not registered Jul 07 20:01:31.263131 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:31.263040 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92ea0d19-f83c-4b0d-969d-cf288f52c55a-kube-api-access-spwbq podName:92ea0d19-f83c-4b0d-969d-cf288f52c55a nodeName:}" failed. No retries permitted until 2026-07-07 20:01:39.263024946 +0000 UTC m=+19.417486881 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-spwbq" (UniqueName: "kubernetes.io/projected/92ea0d19-f83c-4b0d-969d-cf288f52c55a-kube-api-access-spwbq") pod "network-check-target-wj7w5" (UID: "92ea0d19-f83c-4b0d-969d-cf288f52c55a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 20:01:31.679846 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:31.679767 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:31.680031 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:31.679881 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr7c2" podUID="17142129-e574-4092-bd8d-0ffa3de6507a" Jul 07 20:01:31.680031 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:31.679949 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:01:31.680152 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:31.680063 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvph6" podUID="da37b4bb-9b02-4491-9d87-c4f56be7c7ad" Jul 07 20:01:31.680152 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:31.680111 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:01:31.680251 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:31.680199 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wj7w5" podUID="92ea0d19-f83c-4b0d-969d-cf288f52c55a" Jul 07 20:01:33.678882 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:33.678845 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:33.679311 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:33.678892 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:01:33.679311 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:33.678845 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:01:33.679311 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:33.678998 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr7c2" podUID="17142129-e574-4092-bd8d-0ffa3de6507a" Jul 07 20:01:33.679311 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:33.679050 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wj7w5" podUID="92ea0d19-f83c-4b0d-969d-cf288f52c55a" Jul 07 20:01:33.679311 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:33.679164 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvph6" podUID="da37b4bb-9b02-4491-9d87-c4f56be7c7ad" Jul 07 20:01:34.745875 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:34.745805 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jlj7d" event={"ID":"2c31e770-b420-42cf-8bf6-a7f8540ed592","Type":"ContainerStarted","Data":"ec7a9ba5801ae5487da15b4d7fea7b32ab9c0ec26b4cb40e793d34b9ca916bd2"} Jul 07 20:01:34.750344 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:34.750290 2565 generic.go:356] "Generic (PLEG): container finished" podID="e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39" containerID="b63171770a0726052ce7e92c8318d2ad5a1ef3a091df51ac697b3507a7da8c01" exitCode=0 Jul 07 20:01:34.750493 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:34.750373 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vx82p" event={"ID":"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39","Type":"ContainerDied","Data":"b63171770a0726052ce7e92c8318d2ad5a1ef3a091df51ac697b3507a7da8c01"} Jul 07 20:01:34.752756 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:34.752733 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b8qtm" event={"ID":"f3290b11-e223-477b-b593-745224204b1e","Type":"ContainerStarted","Data":"0bcb96ca8d07d7d53259ea4432dba09037d88dfe47b1f1ed9376087ac87fde4e"} Jul 07 20:01:34.754915 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:34.754891 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jbxhj" event={"ID":"f5a8c362-f805-466b-ae7b-ca28c4087e63","Type":"ContainerStarted","Data":"6312b3038bf273c3e5609b0a02fdbdb11ed5a1a8ec66f31c6c65c17cf8ed9fbc"} Jul 07 20:01:34.756446 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:34.756428 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" event={"ID":"19ef0176-73b8-465c-abda-754d10ddd21f","Type":"ContainerStarted","Data":"02de8ecf69ec645964ae656b7d9ee9bddaa1c3f6843f2b79da50975f6744b3df"} Jul 07 20:01:34.758082 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:34.758057 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-229.ec2.internal" event={"ID":"59bf4ed301a47b79107844633c8ed6ad","Type":"ContainerStarted","Data":"277d6a258cfa643037f495d5c428908b3ceedf26f19012a3b7df4437b95f6814"} Jul 07 20:01:34.759530 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:34.759511 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfbfw" event={"ID":"dc753a68-481d-469b-88a3-37bbb48ac522","Type":"ContainerStarted","Data":"1d1b4360dcb20595ed3a6d2b697f11d8c257c6d55a13b2e5e3915ec296141619"} Jul 07 20:01:34.768009 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:34.767817 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-229.ec2.internal" podStartSLOduration=12.767801608 podStartE2EDuration="12.767801608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 20:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:01:25.745600987 +0000 UTC m=+5.900062937" watchObservedRunningTime="2026-07-07 20:01:34.767801608 +0000 UTC m=+14.922263558" Jul 07 20:01:34.768447 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:34.768422 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jlj7d" podStartSLOduration=3.7475796260000003 podStartE2EDuration="13.768394216s" totalImagesPullingTime="10.02081459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:01:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:01:34.767739785 +0000 UTC m=+14.922201734" watchObservedRunningTime="2026-07-07 20:01:34.768394216 +0000 UTC m=+14.922856166" Jul 07 20:01:34.795893 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:34.795847 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7gbnw" podStartSLOduration=3.735013487 podStartE2EDuration="13.79582863s" totalImagesPullingTime="10.060815143s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 20:01:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:01:34.79493387 +0000 UTC m=+14.949395819" watchObservedRunningTime="2026-07-07 20:01:34.79582863 +0000 UTC m=+14.950290582" Jul 07 20:01:34.857003 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:34.856949 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-b8qtm" podStartSLOduration=3.8383427980000002 podStartE2EDuration="13.856931198s" totalImagesPullingTime="10.0185884s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 20:01:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:01:34.819854115 +0000 UTC m=+14.974316066" watchObservedRunningTime="2026-07-07 20:01:34.856931198 +0000 UTC m=+15.011393148" Jul 07 20:01:34.881496 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:34.881450 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jbxhj" podStartSLOduration=3.000848525 podStartE2EDuration="12.881436642s" totalImagesPullingTime="9.880588117s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 20:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:01:34.856841694 +0000 UTC m=+15.011303663" watchObservedRunningTime="2026-07-07 20:01:34.881436642 +0000 UTC m=+15.035898591" Jul 07 20:01:34.944603 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:34.944542 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-229.ec2.internal" podStartSLOduration=12.944523622 podStartE2EDuration="12.944523622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 20:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:01:34.890635093 +0000 UTC m=+15.045097042" watchObservedRunningTime="2026-07-07 20:01:34.944523622 +0000 UTC m=+15.098985572" Jul 07 20:01:35.678892 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:35.678862 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:01:35.679058 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:35.678870 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:35.679058 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:35.678990 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:01:35.679058 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:35.678987 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wj7w5" podUID="92ea0d19-f83c-4b0d-969d-cf288f52c55a" Jul 07 20:01:35.679222 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:35.679080 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr7c2" podUID="17142129-e574-4092-bd8d-0ffa3de6507a" Jul 07 20:01:35.679222 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:35.679153 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvph6" podUID="da37b4bb-9b02-4491-9d87-c4f56be7c7ad" Jul 07 20:01:35.763829 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:35.763763 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v6vf5" event={"ID":"dc031eff-d6a1-44f6-803b-bc0fb0341bc0","Type":"ContainerStarted","Data":"92b7616b30704bbe1e10c976ebbae5272cf1889547216298c576dc5fecb40956"} Jul 07 20:01:35.785119 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:35.785068 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-v6vf5" podStartSLOduration=4.773378497 podStartE2EDuration="14.785052376s" totalImagesPullingTime="10.011673879s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 20:01:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:01:35.784389805 +0000 UTC m=+15.938851754" watchObservedRunningTime="2026-07-07 20:01:35.785052376 +0000 UTC m=+15.939514327" Jul 07 20:01:35.960899 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:35.960870 2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 07 20:01:36.549338 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:36.549223 2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T20:01:35.960887956Z","UUID":"a21d3349-5600-4d76-825a-153612705433","Handler":null,"Name":"","Endpoint":""} Jul 07 20:01:36.551635 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:36.551082 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 07 20:01:36.551635 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:36.551127 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 07 20:01:36.768811 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:36.768695 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfbfw" event={"ID":"dc753a68-481d-469b-88a3-37bbb48ac522","Type":"ContainerStarted","Data":"17fb3d98bacbd33f4d47daabf0fa8d180ec7db0809179529d51bd7fa81484bf4"} Jul 07 20:01:37.678964 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:37.678933 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:01:37.679145 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:37.678933 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:37.679145 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:37.679061 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wj7w5" podUID="92ea0d19-f83c-4b0d-969d-cf288f52c55a" Jul 07 20:01:37.679145 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:37.679084 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:01:37.679286 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:37.679193 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr7c2" podUID="17142129-e574-4092-bd8d-0ffa3de6507a" Jul 07 20:01:37.679339 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:37.679287 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvph6" podUID="da37b4bb-9b02-4491-9d87-c4f56be7c7ad" Jul 07 20:01:38.781985 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:38.781951 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jlj7d" Jul 07 20:01:38.782704 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:38.782679 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jlj7d" Jul 07 20:01:39.225840 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:39.225802 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da37b4bb-9b02-4491-9d87-c4f56be7c7ad-metrics-certs\") pod \"network-metrics-daemon-qvph6\" (UID: \"da37b4bb-9b02-4491-9d87-c4f56be7c7ad\") " pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:01:39.226022 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:39.225928 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 20:01:39.226022 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:39.225985 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da37b4bb-9b02-4491-9d87-c4f56be7c7ad-metrics-certs podName:da37b4bb-9b02-4491-9d87-c4f56be7c7ad nodeName:}" failed. No retries permitted until 2026-07-07 20:01:55.225966536 +0000 UTC m=+35.380428464 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da37b4bb-9b02-4491-9d87-c4f56be7c7ad-metrics-certs") pod "network-metrics-daemon-qvph6" (UID: "da37b4bb-9b02-4491-9d87-c4f56be7c7ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 20:01:39.326910 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:39.326880 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17142129-e574-4092-bd8d-0ffa3de6507a-original-pull-secret\") pod \"global-pull-secret-syncer-zr7c2\" (UID: \"17142129-e574-4092-bd8d-0ffa3de6507a\") " pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:39.327067 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:39.326922 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spwbq\" (UniqueName: \"kubernetes.io/projected/92ea0d19-f83c-4b0d-969d-cf288f52c55a-kube-api-access-spwbq\") pod \"network-check-target-wj7w5\" (UID: \"92ea0d19-f83c-4b0d-969d-cf288f52c55a\") " pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:01:39.327067 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:39.327050 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 20:01:39.327154 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:39.327068 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 20:01:39.327154 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:39.327085 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 20:01:39.327154 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:39.327094 2565 projected.go:196] Error preparing data for projected volume kube-api-access-spwbq for pod openshift-network-diagnostics/network-check-target-wj7w5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 20:01:39.327154 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:39.327129 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17142129-e574-4092-bd8d-0ffa3de6507a-original-pull-secret podName:17142129-e574-4092-bd8d-0ffa3de6507a nodeName:}" failed. No retries permitted until 2026-07-07 20:01:55.327108341 +0000 UTC m=+35.481570284 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17142129-e574-4092-bd8d-0ffa3de6507a-original-pull-secret") pod "global-pull-secret-syncer-zr7c2" (UID: "17142129-e574-4092-bd8d-0ffa3de6507a") : object "kube-system"/"original-pull-secret" not registered Jul 07 20:01:39.327154 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:39.327149 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92ea0d19-f83c-4b0d-969d-cf288f52c55a-kube-api-access-spwbq podName:92ea0d19-f83c-4b0d-969d-cf288f52c55a nodeName:}" failed. No retries permitted until 2026-07-07 20:01:55.327138718 +0000 UTC m=+35.481600650 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-spwbq" (UniqueName: "kubernetes.io/projected/92ea0d19-f83c-4b0d-969d-cf288f52c55a-kube-api-access-spwbq") pod "network-check-target-wj7w5" (UID: "92ea0d19-f83c-4b0d-969d-cf288f52c55a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 20:01:39.679090 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:39.679062 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:01:39.679090 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:39.679087 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:01:39.679309 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:39.679092 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:39.679309 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:39.679199 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvph6" podUID="da37b4bb-9b02-4491-9d87-c4f56be7c7ad" Jul 07 20:01:39.679309 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:39.679285 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wj7w5" podUID="92ea0d19-f83c-4b0d-969d-cf288f52c55a" Jul 07 20:01:39.679466 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:39.679351 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr7c2" podUID="17142129-e574-4092-bd8d-0ffa3de6507a" Jul 07 20:01:39.773851 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:39.773820 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jlj7d" Jul 07 20:01:39.775074 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:39.775052 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jlj7d" Jul 07 20:01:41.679629 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:41.679418 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:01:41.680072 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:41.679492 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:01:41.680072 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:41.679703 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvph6" podUID="da37b4bb-9b02-4491-9d87-c4f56be7c7ad" Jul 07 20:01:41.680072 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:41.679528 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:41.680072 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:41.679800 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wj7w5" podUID="92ea0d19-f83c-4b0d-969d-cf288f52c55a" Jul 07 20:01:41.680072 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:41.679876 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr7c2" podUID="17142129-e574-4092-bd8d-0ffa3de6507a" Jul 07 20:01:43.678922 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:43.678885 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:43.679512 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:43.678886 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:01:43.679512 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:43.678885 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:01:43.679512 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:43.679092 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr7c2" podUID="17142129-e574-4092-bd8d-0ffa3de6507a" Jul 07 20:01:43.679512 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:43.679167 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wj7w5" podUID="92ea0d19-f83c-4b0d-969d-cf288f52c55a" Jul 07 20:01:43.679512 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:43.679271 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvph6" podUID="da37b4bb-9b02-4491-9d87-c4f56be7c7ad" Jul 07 20:01:45.679043 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:45.679010 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:01:45.679598 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:45.679010 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:01:45.679598 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:45.679135 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvph6" podUID="da37b4bb-9b02-4491-9d87-c4f56be7c7ad" Jul 07 20:01:45.679598 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:45.679016 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:45.679598 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:45.679212 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wj7w5" podUID="92ea0d19-f83c-4b0d-969d-cf288f52c55a" Jul 07 20:01:45.679598 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:45.679268 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr7c2" podUID="17142129-e574-4092-bd8d-0ffa3de6507a" Jul 07 20:01:47.679809 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:47.679630 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:01:47.680470 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:47.679627 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:47.680470 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:47.679880 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wj7w5" podUID="92ea0d19-f83c-4b0d-969d-cf288f52c55a" Jul 07 20:01:47.680470 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:47.679971 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr7c2" podUID="17142129-e574-4092-bd8d-0ffa3de6507a" Jul 07 20:01:47.680470 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:47.679634 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:01:47.680470 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:47.680100 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvph6" podUID="da37b4bb-9b02-4491-9d87-c4f56be7c7ad" Jul 07 20:01:47.791285 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:47.791255 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" event={"ID":"7f34b812-e3ff-47b7-ad26-092401121393","Type":"ContainerStarted","Data":"f4fb6363509066a6064d33c6488b8830c81bc708ca8068f94808766ffac42ce9"} Jul 07 20:01:47.791471 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:47.791343 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" event={"ID":"7f34b812-e3ff-47b7-ad26-092401121393","Type":"ContainerStarted","Data":"af74e1a1964275e18e76182b1bd3265d444c61df463c32e29b0dfbd7f6e47df4"} Jul 07 20:01:47.791471 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:47.791395 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" event={"ID":"7f34b812-e3ff-47b7-ad26-092401121393","Type":"ContainerStarted","Data":"4716ea5a9c0ccb52d940b2d2a6d9c1179ce3fc2369652311c93b55488b7b5fdd"} Jul 07 20:01:47.791471 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:47.791460 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" event={"ID":"7f34b812-e3ff-47b7-ad26-092401121393","Type":"ContainerStarted","Data":"0537961db93582051c7c011e272fe12cc779a06b610a8a37b1a349a169d6eaf5"} Jul 07 20:01:47.791611 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:47.791506 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" event={"ID":"7f34b812-e3ff-47b7-ad26-092401121393","Type":"ContainerStarted","Data":"ff1a3ca7e88eed2cd1bfcdb91f29163c01a217a37c5334a377e6e8c9e16617ca"} Jul 07 20:01:47.791611 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:47.791539 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" event={"ID":"7f34b812-e3ff-47b7-ad26-092401121393","Type":"ContainerStarted","Data":"1809e66f2be5b71750978c37c84a5ed707b572cc27c3b23500ea2dd36c78b9b3"} Jul 07 20:01:47.793203 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:47.793176 2565 generic.go:356] "Generic (PLEG): container finished" podID="e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39" containerID="4724b07f382496ab4cdda3d39033d8c39ec5b9c0c316a25b8a104cb3aaf075da" exitCode=0 Jul 07 20:01:47.793299 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:47.793250 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vx82p" event={"ID":"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39","Type":"ContainerDied","Data":"4724b07f382496ab4cdda3d39033d8c39ec5b9c0c316a25b8a104cb3aaf075da"} Jul 07 20:01:47.795054 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:47.794730 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-25ms9" event={"ID":"1b90c45b-0580-4343-b306-5bcc726279a6","Type":"ContainerStarted","Data":"2f0461f180397d12d2c763d8dcf9802db71e5e6cf29b22e1e32f50227aa96795"} Jul 07 20:01:47.796831 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:47.796813 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfbfw" event={"ID":"dc753a68-481d-469b-88a3-37bbb48ac522","Type":"ContainerStarted","Data":"9db349f0cd40ae41422126bc36bbbb97eae495065555b3aa681e05d7edd245ec"} Jul 07 20:01:47.847693 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:47.847648 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-25ms9" podStartSLOduration=3.158042086 podStartE2EDuration="25.847633902s" totalImagesPullingTime="22.689591816s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 20:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:01:47.847104097 +0000 UTC m=+28.001566046" watchObservedRunningTime="2026-07-07 20:01:47.847633902 +0000 UTC m=+28.002095850" Jul 07 20:01:49.681699 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:49.681672 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:01:49.682133 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:49.681672 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:01:49.682133 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:49.681793 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wj7w5" podUID="92ea0d19-f83c-4b0d-969d-cf288f52c55a" Jul 07 20:01:49.682133 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:49.681672 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:49.682133 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:49.681839 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvph6" podUID="da37b4bb-9b02-4491-9d87-c4f56be7c7ad" Jul 07 20:01:49.682133 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:49.681891 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr7c2" podUID="17142129-e574-4092-bd8d-0ffa3de6507a" Jul 07 20:01:50.804456 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:50.804418 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" event={"ID":"7f34b812-e3ff-47b7-ad26-092401121393","Type":"ContainerStarted","Data":"06a42e0835864f7c3197ff3b3dd2f1a08b2977777279340d9791dada5401f162"} Jul 07 20:01:50.806481 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:50.806448 2565 generic.go:356] "Generic (PLEG): container finished" podID="e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39" containerID="3e5a40015000ad0ac32842997a5af0894257613e00d3be048f28643a7ff3aa7a" exitCode=0 Jul 07 20:01:50.806593 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:50.806501 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vx82p" event={"ID":"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39","Type":"ContainerDied","Data":"3e5a40015000ad0ac32842997a5af0894257613e00d3be048f28643a7ff3aa7a"} Jul 07 20:01:50.839572 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:50.839526 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfbfw" podStartSLOduration=7.526258938 podStartE2EDuration="29.839514068s" totalImagesPullingTime="22.31325513s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 20:01:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:01:47.870538821 +0000 UTC m=+28.025000769" watchObservedRunningTime="2026-07-07 20:01:50.839514068 +0000 UTC m=+30.993976051" Jul 07 20:01:51.497880 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:51.497854 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b8qtm_f3290b11-e223-477b-b593-745224204b1e/dns-node-resolver/0.log" Jul 07 20:01:51.680323 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:51.680288 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:51.680477 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:51.680376 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:01:51.680477 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:51.680415 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:01:51.680477 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:51.680428 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr7c2" podUID="17142129-e574-4092-bd8d-0ffa3de6507a" Jul 07 20:01:51.680581 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:51.680474 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvph6" podUID="da37b4bb-9b02-4491-9d87-c4f56be7c7ad" Jul 07 20:01:51.680620 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:51.680571 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wj7w5" podUID="92ea0d19-f83c-4b0d-969d-cf288f52c55a" Jul 07 20:01:52.446718 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:52.446547 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jbxhj_f5a8c362-f805-466b-ae7b-ca28c4087e63/node-ca/0.log" Jul 07 20:01:52.815910 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:52.815824 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" event={"ID":"7f34b812-e3ff-47b7-ad26-092401121393","Type":"ContainerStarted","Data":"47cf2c4eb0c39ca017310cb7e96fe7c9029e80ba5129c6c32149d4acc85c8a9f"} Jul 07 20:01:52.816166 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:52.816125 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:52.816290 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:52.816197 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:52.816290 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:52.816245 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:52.818627 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:52.818597 2565 generic.go:356] "Generic (PLEG): container finished" podID="e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39" containerID="b66f6b03b33f2e96a90119eb9ab4925f6c201f7f836c13f5bcc30714b843ebfc" exitCode=0 Jul 07 20:01:52.818732 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:52.818641 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vx82p" event={"ID":"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39","Type":"ContainerDied","Data":"b66f6b03b33f2e96a90119eb9ab4925f6c201f7f836c13f5bcc30714b843ebfc"} Jul 07 20:01:52.831422 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:52.831389 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:52.832001 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:52.831987 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:01:52.858873 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:52.858835 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" podStartSLOduration=8.363329595 podStartE2EDuration="30.858819833s" totalImagesPullingTime="22.495490238s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 20:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:01:52.858712835 +0000 UTC m=+33.013174785" watchObservedRunningTime="2026-07-07 20:01:52.858819833 +0000 UTC m=+33.013281782" Jul 07 20:01:53.679060 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:53.679033 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:53.679558 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:53.679030 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:01:53.679558 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:53.679147 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr7c2" podUID="17142129-e574-4092-bd8d-0ffa3de6507a" Jul 07 20:01:53.679558 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:53.679031 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:01:53.679558 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:53.679218 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvph6" podUID="da37b4bb-9b02-4491-9d87-c4f56be7c7ad" Jul 07 20:01:53.679558 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:53.679317 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wj7w5" podUID="92ea0d19-f83c-4b0d-969d-cf288f52c55a" Jul 07 20:01:54.015725 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:54.015646 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wj7w5"] Jul 07 20:01:54.015874 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:54.015734 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:01:54.015874 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:54.015807 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wj7w5" podUID="92ea0d19-f83c-4b0d-969d-cf288f52c55a" Jul 07 20:01:54.019091 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:54.019067 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qvph6"] Jul 07 20:01:54.019217 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:54.019159 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:01:54.019305 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:54.019263 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvph6" podUID="da37b4bb-9b02-4491-9d87-c4f56be7c7ad" Jul 07 20:01:54.019800 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:54.019771 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zr7c2"] Jul 07 20:01:54.019923 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:54.019858 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:54.019982 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:54.019936 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr7c2" podUID="17142129-e574-4092-bd8d-0ffa3de6507a" Jul 07 20:01:55.242597 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:55.242525 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da37b4bb-9b02-4491-9d87-c4f56be7c7ad-metrics-certs\") pod \"network-metrics-daemon-qvph6\" (UID: \"da37b4bb-9b02-4491-9d87-c4f56be7c7ad\") " pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:01:55.243048 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:55.242705 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 20:01:55.243048 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:55.242783 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da37b4bb-9b02-4491-9d87-c4f56be7c7ad-metrics-certs podName:da37b4bb-9b02-4491-9d87-c4f56be7c7ad nodeName:}" failed. No retries permitted until 2026-07-07 20:02:27.242761341 +0000 UTC m=+67.397223271 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da37b4bb-9b02-4491-9d87-c4f56be7c7ad-metrics-certs") pod "network-metrics-daemon-qvph6" (UID: "da37b4bb-9b02-4491-9d87-c4f56be7c7ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 20:01:55.343911 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:55.343881 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17142129-e574-4092-bd8d-0ffa3de6507a-original-pull-secret\") pod \"global-pull-secret-syncer-zr7c2\" (UID: \"17142129-e574-4092-bd8d-0ffa3de6507a\") " pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:55.344074 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:55.343923 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spwbq\" (UniqueName: \"kubernetes.io/projected/92ea0d19-f83c-4b0d-969d-cf288f52c55a-kube-api-access-spwbq\") pod \"network-check-target-wj7w5\" (UID: \"92ea0d19-f83c-4b0d-969d-cf288f52c55a\") " pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:01:55.344074 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:55.344038 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 20:01:55.344074 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:55.344049 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 20:01:55.344074 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:55.344068 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 20:01:55.344250 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:55.344080 2565 projected.go:196] Error preparing data for projected volume kube-api-access-spwbq for pod openshift-network-diagnostics/network-check-target-wj7w5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 20:01:55.344250 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:55.344099 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17142129-e574-4092-bd8d-0ffa3de6507a-original-pull-secret podName:17142129-e574-4092-bd8d-0ffa3de6507a nodeName:}" failed. No retries permitted until 2026-07-07 20:02:27.344084216 +0000 UTC m=+67.498546143 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17142129-e574-4092-bd8d-0ffa3de6507a-original-pull-secret") pod "global-pull-secret-syncer-zr7c2" (UID: "17142129-e574-4092-bd8d-0ffa3de6507a") : object "kube-system"/"original-pull-secret" not registered Jul 07 20:01:55.344250 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:55.344127 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92ea0d19-f83c-4b0d-969d-cf288f52c55a-kube-api-access-spwbq podName:92ea0d19-f83c-4b0d-969d-cf288f52c55a nodeName:}" failed. No retries permitted until 2026-07-07 20:02:27.344113121 +0000 UTC m=+67.498575057 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-spwbq" (UniqueName: "kubernetes.io/projected/92ea0d19-f83c-4b0d-969d-cf288f52c55a-kube-api-access-spwbq") pod "network-check-target-wj7w5" (UID: "92ea0d19-f83c-4b0d-969d-cf288f52c55a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 20:01:55.678910 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:55.678883 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:01:55.678910 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:55.678911 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:55.679163 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:55.678884 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:01:55.679163 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:55.678982 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wj7w5" podUID="92ea0d19-f83c-4b0d-969d-cf288f52c55a" Jul 07 20:01:55.679163 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:55.679089 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvph6" podUID="da37b4bb-9b02-4491-9d87-c4f56be7c7ad" Jul 07 20:01:55.679273 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:55.679176 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr7c2" podUID="17142129-e574-4092-bd8d-0ffa3de6507a" Jul 07 20:01:57.679073 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:57.679040 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:01:57.679553 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:57.679040 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:01:57.679553 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:57.679139 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wj7w5" podUID="92ea0d19-f83c-4b0d-969d-cf288f52c55a" Jul 07 20:01:57.679553 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:57.679157 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:57.679553 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:57.679277 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvph6" podUID="da37b4bb-9b02-4491-9d87-c4f56be7c7ad" Jul 07 20:01:57.679553 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:57.679455 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr7c2" podUID="17142129-e574-4092-bd8d-0ffa3de6507a" Jul 07 20:01:59.679411 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:59.679368 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:01:59.680044 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:59.679368 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:01:59.680044 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:59.679499 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvph6" podUID="da37b4bb-9b02-4491-9d87-c4f56be7c7ad" Jul 07 20:01:59.680044 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:59.679595 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr7c2" podUID="17142129-e574-4092-bd8d-0ffa3de6507a" Jul 07 20:01:59.680044 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:01:59.679654 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:01:59.680044 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:01:59.679736 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wj7w5" podUID="92ea0d19-f83c-4b0d-969d-cf288f52c55a" Jul 07 20:02:01.680659 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:01.680438 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:02:01.681244 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:01.680461 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:02:01.681244 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:01.680764 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvph6" podUID="da37b4bb-9b02-4491-9d87-c4f56be7c7ad" Jul 07 20:02:01.681244 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:01.680483 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:02:01.681244 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:01.680841 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wj7w5" podUID="92ea0d19-f83c-4b0d-969d-cf288f52c55a" Jul 07 20:02:01.681244 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:01.680907 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr7c2" podUID="17142129-e574-4092-bd8d-0ffa3de6507a" Jul 07 20:02:03.678996 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:03.678962 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:02:03.679624 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:03.678972 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:02:03.679624 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:03.679082 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr7c2" podUID="17142129-e574-4092-bd8d-0ffa3de6507a" Jul 07 20:02:03.679624 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:03.679110 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:02:03.679624 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:03.679189 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvph6" podUID="da37b4bb-9b02-4491-9d87-c4f56be7c7ad" Jul 07 20:02:03.679624 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:03.679234 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wj7w5" podUID="92ea0d19-f83c-4b0d-969d-cf288f52c55a" Jul 07 20:02:03.848366 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:03.848284 2565 generic.go:356] "Generic (PLEG): container finished" podID="e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39" containerID="ecabd123bb4e4108e5c39c1839e8d54df1d1efb4d2c88c978a48c521b484859b" exitCode=0 Jul 07 20:02:03.848366 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:03.848344 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vx82p" event={"ID":"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39","Type":"ContainerDied","Data":"ecabd123bb4e4108e5c39c1839e8d54df1d1efb4d2c88c978a48c521b484859b"} Jul 07 20:02:04.853618 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:04.853582 2565 generic.go:356] "Generic (PLEG): container finished" podID="e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39" containerID="52e800a4a494a70b278ac8ee3d4ca97c236a0dc3eb25a5927ecb357172aa451f" exitCode=0 Jul 07 20:02:04.854109 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:04.853630 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vx82p" event={"ID":"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39","Type":"ContainerDied","Data":"52e800a4a494a70b278ac8ee3d4ca97c236a0dc3eb25a5927ecb357172aa451f"} Jul 07 20:02:05.679176 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:05.679145 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:02:05.679366 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:05.679262 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wj7w5" podUID="92ea0d19-f83c-4b0d-969d-cf288f52c55a" Jul 07 20:02:05.679366 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:05.679322 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:02:05.679501 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:05.679433 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvph6" podUID="da37b4bb-9b02-4491-9d87-c4f56be7c7ad" Jul 07 20:02:05.679501 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:05.679482 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:02:05.679619 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:05.679599 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr7c2" podUID="17142129-e574-4092-bd8d-0ffa3de6507a" Jul 07 20:02:05.859849 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:05.859816 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vx82p" event={"ID":"e1065bc7-1b4f-4a1f-80e8-6c75f45f9b39","Type":"ContainerStarted","Data":"22e4fd0607fd5db801c23bae56ae8bfacc3e8046c2cbc3bd3a9cf1bc97dae1ca"} Jul 07 20:02:05.897267 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:05.897211 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vx82p" podStartSLOduration=9.995696245 podStartE2EDuration="44.897194109s" totalImagesPullingTime="34.901497864s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 20:01:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:02:05.896605233 +0000 UTC m=+46.051067181" watchObservedRunningTime="2026-07-07 20:02:05.897194109 +0000 UTC m=+46.051656059" Jul 07 20:02:07.679163 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:07.679132 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:02:07.679163 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:07.679132 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:02:07.679593 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:07.679260 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:02:07.679593 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:07.679263 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr7c2" podUID="17142129-e574-4092-bd8d-0ffa3de6507a" Jul 07 20:02:07.679593 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:07.679360 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wj7w5" podUID="92ea0d19-f83c-4b0d-969d-cf288f52c55a" Jul 07 20:02:07.679593 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:07.679476 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvph6" podUID="da37b4bb-9b02-4491-9d87-c4f56be7c7ad" Jul 07 20:02:09.679105 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:09.679069 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:02:09.679507 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:09.679073 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:02:09.679507 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:09.679190 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvph6" podUID="da37b4bb-9b02-4491-9d87-c4f56be7c7ad" Jul 07 20:02:09.679507 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:09.679276 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wj7w5" podUID="92ea0d19-f83c-4b0d-969d-cf288f52c55a" Jul 07 20:02:09.679507 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:09.679078 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:02:09.679507 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:09.679379 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr7c2" podUID="17142129-e574-4092-bd8d-0ffa3de6507a" Jul 07 20:02:11.680246 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:11.680214 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:02:11.680665 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:11.680302 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr7c2" podUID="17142129-e574-4092-bd8d-0ffa3de6507a" Jul 07 20:02:11.680665 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:11.680414 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:02:11.680665 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:11.680634 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvph6" podUID="da37b4bb-9b02-4491-9d87-c4f56be7c7ad" Jul 07 20:02:11.680777 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:11.680667 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:02:11.680777 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:11.680724 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wj7w5" podUID="92ea0d19-f83c-4b0d-969d-cf288f52c55a" Jul 07 20:02:13.679599 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:13.679574 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:02:13.679599 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:13.679599 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:02:13.680068 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:13.679579 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:02:13.680068 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:13.679673 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr7c2" podUID="17142129-e574-4092-bd8d-0ffa3de6507a" Jul 07 20:02:13.680068 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:13.679795 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvph6" podUID="da37b4bb-9b02-4491-9d87-c4f56be7c7ad" Jul 07 20:02:13.680068 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:13.679836 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wj7w5" podUID="92ea0d19-f83c-4b0d-969d-cf288f52c55a" Jul 07 20:02:15.679485 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:15.679451 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:02:15.679971 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:15.679551 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvph6" podUID="da37b4bb-9b02-4491-9d87-c4f56be7c7ad" Jul 07 20:02:15.679971 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:15.679583 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:02:15.679971 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:15.679702 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:02:15.679971 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:15.679699 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr7c2" podUID="17142129-e574-4092-bd8d-0ffa3de6507a" Jul 07 20:02:15.679971 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:15.679757 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wj7w5" podUID="92ea0d19-f83c-4b0d-969d-cf288f52c55a" Jul 07 20:02:17.678985 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:17.678952 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:02:17.678985 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:17.678984 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:02:17.679512 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:17.679054 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wj7w5" podUID="92ea0d19-f83c-4b0d-969d-cf288f52c55a" Jul 07 20:02:17.679512 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:17.679100 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr7c2" podUID="17142129-e574-4092-bd8d-0ffa3de6507a" Jul 07 20:02:17.679512 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:17.679154 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:02:17.679512 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:17.679246 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvph6" podUID="da37b4bb-9b02-4491-9d87-c4f56be7c7ad" Jul 07 20:02:18.623818 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.623740 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-229.ec2.internal" event="NodeReady" Jul 07 20:02:18.623982 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.623868 2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 07 20:02:18.741661 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.741629 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2hcpv"] Jul 07 20:02:18.744932 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.744329 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2hcpv" Jul 07 20:02:18.749764 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.749741 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 07 20:02:18.750347 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.750329 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 07 20:02:18.751290 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.751014 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b4fhd\"" Jul 07 20:02:18.751290 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.751068 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 07 20:02:18.751290 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.751113 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 07 20:02:18.769010 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.768991 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2hcpv"] Jul 07 20:02:18.812181 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.812156 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2hcpv\" (UID: \"bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a\") " pod="openshift-insights/insights-runtime-extractor-2hcpv" Jul 07 20:02:18.812297 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.812199 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a-data-volume\") pod \"insights-runtime-extractor-2hcpv\" (UID: \"bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a\") " pod="openshift-insights/insights-runtime-extractor-2hcpv" Jul 07 20:02:18.812297 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.812217 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a-crio-socket\") pod \"insights-runtime-extractor-2hcpv\" (UID: \"bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a\") " pod="openshift-insights/insights-runtime-extractor-2hcpv" Jul 07 20:02:18.812428 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.812333 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2hcpv\" (UID: \"bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a\") " pod="openshift-insights/insights-runtime-extractor-2hcpv" Jul 07 20:02:18.812428 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.812374 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hvvc\" (UniqueName: \"kubernetes.io/projected/bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a-kube-api-access-4hvvc\") pod \"insights-runtime-extractor-2hcpv\" (UID: \"bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a\") " pod="openshift-insights/insights-runtime-extractor-2hcpv" Jul 07 20:02:18.837800 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.837781 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4wwvr"] Jul 07 20:02:18.839575 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.839562 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-87llx"] Jul 07 20:02:18.839695 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.839679 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4wwvr" Jul 07 20:02:18.841500 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.841482 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-87llx" Jul 07 20:02:18.842935 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.842915 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s6xh9\"" Jul 07 20:02:18.843035 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.842960 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 07 20:02:18.843235 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.843216 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 07 20:02:18.844670 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.844653 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 07 20:02:18.846768 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.846752 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 07 20:02:18.846847 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.846778 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 07 20:02:18.847431 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.847415 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-52pqs\"" Jul 07 20:02:18.852704 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.852685 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-87llx"] Jul 07 20:02:18.858289 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.858271 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4wwvr"] Jul 07 20:02:18.913100 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.913077 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2hcpv\" (UID: \"bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a\") " pod="openshift-insights/insights-runtime-extractor-2hcpv" Jul 07 20:02:18.913240 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.913111 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hvvc\" (UniqueName: \"kubernetes.io/projected/bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a-kube-api-access-4hvvc\") pod \"insights-runtime-extractor-2hcpv\" (UID: \"bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a\") " pod="openshift-insights/insights-runtime-extractor-2hcpv" Jul 07 20:02:18.913240 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.913140 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b7e3a9b-4519-44fd-93c2-60850dca6128-tmp-dir\") pod \"dns-default-4wwvr\" (UID: \"4b7e3a9b-4519-44fd-93c2-60850dca6128\") " pod="openshift-dns/dns-default-4wwvr" Jul 07 20:02:18.913240 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.913178 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2hcpv\" (UID: \"bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a\") " pod="openshift-insights/insights-runtime-extractor-2hcpv" Jul 07 20:02:18.913240 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.913199 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20c78065-0c7e-41b4-91a7-bc885a651f4b-cert\") pod \"ingress-canary-87llx\" (UID: \"20c78065-0c7e-41b4-91a7-bc885a651f4b\") " pod="openshift-ingress-canary/ingress-canary-87llx" Jul 07 20:02:18.913240 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.913223 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b7e3a9b-4519-44fd-93c2-60850dca6128-metrics-tls\") pod \"dns-default-4wwvr\" (UID: \"4b7e3a9b-4519-44fd-93c2-60850dca6128\") " pod="openshift-dns/dns-default-4wwvr" Jul 07 20:02:18.913483 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.913244 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njmcp\" (UniqueName: \"kubernetes.io/projected/20c78065-0c7e-41b4-91a7-bc885a651f4b-kube-api-access-njmcp\") pod \"ingress-canary-87llx\" (UID: \"20c78065-0c7e-41b4-91a7-bc885a651f4b\") " pod="openshift-ingress-canary/ingress-canary-87llx" Jul 07 20:02:18.913483 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.913319 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a-data-volume\") pod \"insights-runtime-extractor-2hcpv\" (UID: \"bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a\") " pod="openshift-insights/insights-runtime-extractor-2hcpv" Jul 07 20:02:18.913483 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.913359 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a-crio-socket\") pod \"insights-runtime-extractor-2hcpv\" (UID: \"bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a\") " pod="openshift-insights/insights-runtime-extractor-2hcpv" Jul 07 20:02:18.913483 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.913440 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4b7e3a9b-4519-44fd-93c2-60850dca6128-config-volume\") pod \"dns-default-4wwvr\" (UID: \"4b7e3a9b-4519-44fd-93c2-60850dca6128\") " pod="openshift-dns/dns-default-4wwvr" Jul 07 20:02:18.913483 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.913459 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxp5q\" (UniqueName: \"kubernetes.io/projected/4b7e3a9b-4519-44fd-93c2-60850dca6128-kube-api-access-mxp5q\") pod \"dns-default-4wwvr\" (UID: \"4b7e3a9b-4519-44fd-93c2-60850dca6128\") " pod="openshift-dns/dns-default-4wwvr" Jul 07 20:02:18.913483 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.913473 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a-crio-socket\") pod \"insights-runtime-extractor-2hcpv\" (UID: \"bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a\") " pod="openshift-insights/insights-runtime-extractor-2hcpv" Jul 07 20:02:18.913741 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.913650 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a-data-volume\") pod \"insights-runtime-extractor-2hcpv\" (UID: \"bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a\") " pod="openshift-insights/insights-runtime-extractor-2hcpv" Jul 07 20:02:18.913741 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.913685 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2hcpv\" (UID: \"bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a\") " pod="openshift-insights/insights-runtime-extractor-2hcpv" Jul 07 20:02:18.917262 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.917244 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2hcpv\" (UID: \"bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a\") " pod="openshift-insights/insights-runtime-extractor-2hcpv" Jul 07 20:02:18.925447 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:18.925426 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hvvc\" (UniqueName: \"kubernetes.io/projected/bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a-kube-api-access-4hvvc\") pod \"insights-runtime-extractor-2hcpv\" (UID: \"bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a\") " pod="openshift-insights/insights-runtime-extractor-2hcpv" Jul 07 20:02:19.014350 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:19.014324 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4b7e3a9b-4519-44fd-93c2-60850dca6128-config-volume\") pod \"dns-default-4wwvr\" (UID: \"4b7e3a9b-4519-44fd-93c2-60850dca6128\") " pod="openshift-dns/dns-default-4wwvr" Jul 07 20:02:19.014504 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:19.014356 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxp5q\" (UniqueName: \"kubernetes.io/projected/4b7e3a9b-4519-44fd-93c2-60850dca6128-kube-api-access-mxp5q\") pod \"dns-default-4wwvr\" (UID: \"4b7e3a9b-4519-44fd-93c2-60850dca6128\") " pod="openshift-dns/dns-default-4wwvr" Jul 07 20:02:19.014504 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:19.014394 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b7e3a9b-4519-44fd-93c2-60850dca6128-tmp-dir\") pod \"dns-default-4wwvr\" (UID: \"4b7e3a9b-4519-44fd-93c2-60850dca6128\") " pod="openshift-dns/dns-default-4wwvr" Jul 07 20:02:19.014504 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:19.014449 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20c78065-0c7e-41b4-91a7-bc885a651f4b-cert\") pod \"ingress-canary-87llx\" (UID: \"20c78065-0c7e-41b4-91a7-bc885a651f4b\") " pod="openshift-ingress-canary/ingress-canary-87llx" Jul 07 20:02:19.014504 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:19.014473 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b7e3a9b-4519-44fd-93c2-60850dca6128-metrics-tls\") pod \"dns-default-4wwvr\" (UID: \"4b7e3a9b-4519-44fd-93c2-60850dca6128\") " pod="openshift-dns/dns-default-4wwvr" Jul 07 20:02:19.014504 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:19.014494 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njmcp\" (UniqueName: \"kubernetes.io/projected/20c78065-0c7e-41b4-91a7-bc885a651f4b-kube-api-access-njmcp\") pod \"ingress-canary-87llx\" (UID: \"20c78065-0c7e-41b4-91a7-bc885a651f4b\") " pod="openshift-ingress-canary/ingress-canary-87llx" Jul 07 20:02:19.014779 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:19.014759 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b7e3a9b-4519-44fd-93c2-60850dca6128-tmp-dir\") pod \"dns-default-4wwvr\" (UID: \"4b7e3a9b-4519-44fd-93c2-60850dca6128\") " pod="openshift-dns/dns-default-4wwvr" Jul 07 20:02:19.014945 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:19.014916 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4b7e3a9b-4519-44fd-93c2-60850dca6128-config-volume\") pod \"dns-default-4wwvr\" (UID: \"4b7e3a9b-4519-44fd-93c2-60850dca6128\") " pod="openshift-dns/dns-default-4wwvr" Jul 07 20:02:19.016640 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:19.016617 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4b7e3a9b-4519-44fd-93c2-60850dca6128-metrics-tls\") pod \"dns-default-4wwvr\" (UID: \"4b7e3a9b-4519-44fd-93c2-60850dca6128\") " pod="openshift-dns/dns-default-4wwvr" Jul 07 20:02:19.016746 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:19.016728 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20c78065-0c7e-41b4-91a7-bc885a651f4b-cert\") pod \"ingress-canary-87llx\" (UID: \"20c78065-0c7e-41b4-91a7-bc885a651f4b\") " pod="openshift-ingress-canary/ingress-canary-87llx" Jul 07 20:02:19.034548 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:19.034517 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxp5q\" (UniqueName: \"kubernetes.io/projected/4b7e3a9b-4519-44fd-93c2-60850dca6128-kube-api-access-mxp5q\") pod \"dns-default-4wwvr\" (UID: \"4b7e3a9b-4519-44fd-93c2-60850dca6128\") " pod="openshift-dns/dns-default-4wwvr" Jul 07 20:02:19.035694 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:19.035672 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njmcp\" (UniqueName: \"kubernetes.io/projected/20c78065-0c7e-41b4-91a7-bc885a651f4b-kube-api-access-njmcp\") pod \"ingress-canary-87llx\" (UID: \"20c78065-0c7e-41b4-91a7-bc885a651f4b\") " pod="openshift-ingress-canary/ingress-canary-87llx" Jul 07 20:02:19.054319 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:19.054297 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2hcpv" Jul 07 20:02:19.148888 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:19.148860 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4wwvr" Jul 07 20:02:19.154050 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:19.154025 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-87llx" Jul 07 20:02:19.219268 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:19.214574 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2hcpv"] Jul 07 20:02:19.235790 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:02:19.235757 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb1f278_7bd1_4f66_b373_dc5b7ebbfd7a.slice/crio-ce515a67bbb3fa95adc2a1d077623eaf89c3c44a38d9a6ab1a49d84df9d9228b WatchSource:0}: Error finding container ce515a67bbb3fa95adc2a1d077623eaf89c3c44a38d9a6ab1a49d84df9d9228b: Status 404 returned error can't find the container with id ce515a67bbb3fa95adc2a1d077623eaf89c3c44a38d9a6ab1a49d84df9d9228b Jul 07 20:02:19.309826 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:19.309798 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4wwvr"] Jul 07 20:02:19.317791 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:02:19.317734 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b7e3a9b_4519_44fd_93c2_60850dca6128.slice/crio-e8b3d2e958e0d9109407f1c958f40a6cf2d4e0c6a35003e0e4827d0e99687a9c WatchSource:0}: Error finding container e8b3d2e958e0d9109407f1c958f40a6cf2d4e0c6a35003e0e4827d0e99687a9c: Status 404 returned error can't find the container with id e8b3d2e958e0d9109407f1c958f40a6cf2d4e0c6a35003e0e4827d0e99687a9c Jul 07 20:02:19.328758 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:19.328737 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-87llx"] Jul 07 20:02:19.331641 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:02:19.331616 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20c78065_0c7e_41b4_91a7_bc885a651f4b.slice/crio-6280a00e6fa0eaa57d999cd045cf37188b7871f8195177a2e6d2ffd0e09926ee WatchSource:0}: Error finding container 6280a00e6fa0eaa57d999cd045cf37188b7871f8195177a2e6d2ffd0e09926ee: Status 404 returned error can't find the container with id 6280a00e6fa0eaa57d999cd045cf37188b7871f8195177a2e6d2ffd0e09926ee Jul 07 20:02:19.679136 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:19.679104 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:02:19.679313 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:19.679104 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:02:19.679313 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:19.679108 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:02:19.681736 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:19.681718 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 20:02:19.682913 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:19.682895 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lcjbq\"" Jul 07 20:02:19.682996 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:19.682895 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 20:02:19.683791 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:19.683745 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 20:02:19.683791 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:19.683757 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mcw2j\"" Jul 07 20:02:19.683791 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:19.683778 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 20:02:19.891092 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:19.891045 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-87llx" event={"ID":"20c78065-0c7e-41b4-91a7-bc885a651f4b","Type":"ContainerStarted","Data":"6280a00e6fa0eaa57d999cd045cf37188b7871f8195177a2e6d2ffd0e09926ee"} Jul 07 20:02:19.892830 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:19.892803 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4wwvr" event={"ID":"4b7e3a9b-4519-44fd-93c2-60850dca6128","Type":"ContainerStarted","Data":"e8b3d2e958e0d9109407f1c958f40a6cf2d4e0c6a35003e0e4827d0e99687a9c"} Jul 07 20:02:19.896391 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:19.896366 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2hcpv" event={"ID":"bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a","Type":"ContainerStarted","Data":"ce515a67bbb3fa95adc2a1d077623eaf89c3c44a38d9a6ab1a49d84df9d9228b"} Jul 07 20:02:22.013500 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:22.013466 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-2vmh4"] Jul 07 20:02:22.024141 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:22.024111 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2vmh4" Jul 07 20:02:22.027533 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:22.027509 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-j976q\"" Jul 07 20:02:22.027670 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:22.027637 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 07 20:02:22.028581 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:22.028562 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 07 20:02:22.028693 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:22.028607 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 07 20:02:22.028797 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:22.028718 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 07 20:02:22.028797 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:22.028729 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 07 20:02:22.033507 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:22.033470 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-2vmh4"] Jul 07 20:02:22.138066 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:22.138019 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mndfg\" (UniqueName: \"kubernetes.io/projected/eb073572-f7a1-40e6-9485-a6206bffda9b-kube-api-access-mndfg\") pod \"prometheus-operator-567f7b8b7c-2vmh4\" (UID: \"eb073572-f7a1-40e6-9485-a6206bffda9b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2vmh4" Jul 07 20:02:22.138233 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:22.138083 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb073572-f7a1-40e6-9485-a6206bffda9b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-2vmh4\" (UID: \"eb073572-f7a1-40e6-9485-a6206bffda9b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2vmh4" Jul 07 20:02:22.138233 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:22.138139 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb073572-f7a1-40e6-9485-a6206bffda9b-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-2vmh4\" (UID: \"eb073572-f7a1-40e6-9485-a6206bffda9b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2vmh4" Jul 07 20:02:22.138233 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:22.138214 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb073572-f7a1-40e6-9485-a6206bffda9b-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-2vmh4\" (UID: \"eb073572-f7a1-40e6-9485-a6206bffda9b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2vmh4" Jul 07 20:02:22.239573 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:22.239532 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mndfg\" (UniqueName: \"kubernetes.io/projected/eb073572-f7a1-40e6-9485-a6206bffda9b-kube-api-access-mndfg\") pod \"prometheus-operator-567f7b8b7c-2vmh4\" (UID: \"eb073572-f7a1-40e6-9485-a6206bffda9b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2vmh4" Jul 07 20:02:22.239763 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:22.239591 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb073572-f7a1-40e6-9485-a6206bffda9b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-2vmh4\" (UID: \"eb073572-f7a1-40e6-9485-a6206bffda9b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2vmh4" Jul 07 20:02:22.239763 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:22.239629 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb073572-f7a1-40e6-9485-a6206bffda9b-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-2vmh4\" (UID: \"eb073572-f7a1-40e6-9485-a6206bffda9b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2vmh4" Jul 07 20:02:22.239763 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:22.239690 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb073572-f7a1-40e6-9485-a6206bffda9b-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-2vmh4\" (UID: \"eb073572-f7a1-40e6-9485-a6206bffda9b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2vmh4" Jul 07 20:02:22.239925 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:22.239805 2565 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 07 20:02:22.239979 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:22.239928 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb073572-f7a1-40e6-9485-a6206bffda9b-prometheus-operator-tls podName:eb073572-f7a1-40e6-9485-a6206bffda9b nodeName:}" failed. No retries permitted until 2026-07-07 20:02:22.739905681 +0000 UTC m=+62.894367626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/eb073572-f7a1-40e6-9485-a6206bffda9b-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-2vmh4" (UID: "eb073572-f7a1-40e6-9485-a6206bffda9b") : secret "prometheus-operator-tls" not found Jul 07 20:02:22.240442 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:22.240416 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb073572-f7a1-40e6-9485-a6206bffda9b-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-2vmh4\" (UID: \"eb073572-f7a1-40e6-9485-a6206bffda9b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2vmh4" Jul 07 20:02:22.242274 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:22.242252 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb073572-f7a1-40e6-9485-a6206bffda9b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-2vmh4\" (UID: \"eb073572-f7a1-40e6-9485-a6206bffda9b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2vmh4" Jul 07 20:02:22.258156 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:22.258124 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mndfg\" (UniqueName: \"kubernetes.io/projected/eb073572-f7a1-40e6-9485-a6206bffda9b-kube-api-access-mndfg\") pod \"prometheus-operator-567f7b8b7c-2vmh4\" (UID: \"eb073572-f7a1-40e6-9485-a6206bffda9b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2vmh4" Jul 07 20:02:22.744018 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:22.743982 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb073572-f7a1-40e6-9485-a6206bffda9b-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-2vmh4\" (UID: \"eb073572-f7a1-40e6-9485-a6206bffda9b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2vmh4" Jul 07 20:02:22.746912 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:22.746872 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/eb073572-f7a1-40e6-9485-a6206bffda9b-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-2vmh4\" (UID: \"eb073572-f7a1-40e6-9485-a6206bffda9b\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2vmh4" Jul 07 20:02:22.935782 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:22.935746 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2vmh4" Jul 07 20:02:24.541523 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:24.540793 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-2vmh4"] Jul 07 20:02:24.833518 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:24.833452 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6skbt" Jul 07 20:02:24.912213 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:24.912169 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-87llx" event={"ID":"20c78065-0c7e-41b4-91a7-bc885a651f4b","Type":"ContainerStarted","Data":"f3fad1a79c8d4adad3596ab9a226237d6338e4e27337fcaae122483b063d481e"} Jul 07 20:02:24.914214 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:24.914176 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4wwvr" event={"ID":"4b7e3a9b-4519-44fd-93c2-60850dca6128","Type":"ContainerStarted","Data":"b73a5b043d789c3b91b77824994d8574f874666493b0dcd71a8806715c0d5ad4"} Jul 07 20:02:24.914339 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:24.914276 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4wwvr" Jul 07 20:02:24.914339 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:24.914308 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4wwvr" event={"ID":"4b7e3a9b-4519-44fd-93c2-60850dca6128","Type":"ContainerStarted","Data":"426baf9f927a1f106a39995637359338addcc481a6de705d856bd6bfc9f0db33"} Jul 07 20:02:24.915742 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:24.915720 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2vmh4" event={"ID":"eb073572-f7a1-40e6-9485-a6206bffda9b","Type":"ContainerStarted","Data":"5f0a2e632ef75f5f8d4f2a97e10e3e5848cba1bc4e8f8c66086c42934a779d80"} Jul 07 20:02:24.917480 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:24.917459 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2hcpv" event={"ID":"bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a","Type":"ContainerStarted","Data":"74f631128559bb90824a6ef923da5b5dcf52dbc8e2439ee0d64ca349ceb8066b"} Jul 07 20:02:24.940602 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:24.940553 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-87llx" podStartSLOduration=1.8777085260000002 podStartE2EDuration="6.940534738s" totalImagesPullingTime="5.062826212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:02:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:02:24.939681148 +0000 UTC m=+65.094143099" watchObservedRunningTime="2026-07-07 20:02:24.940534738 +0000 UTC m=+65.094996668" Jul 07 20:02:24.984541 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:24.981464 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4wwvr" podStartSLOduration=1.904530439 podStartE2EDuration="6.981439262s" totalImagesPullingTime="5.076908823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:02:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:02:24.978769684 +0000 UTC m=+65.133231632" watchObservedRunningTime="2026-07-07 20:02:24.981439262 +0000 UTC m=+65.135901224" Jul 07 20:02:25.921855 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:25.921821 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2hcpv" event={"ID":"bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a","Type":"ContainerStarted","Data":"7a140f821602fc156f1589e2f76e03f49ca6c0e387fb4ccd86ced5aca470a844"} Jul 07 20:02:26.928053 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:26.928020 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2vmh4" event={"ID":"eb073572-f7a1-40e6-9485-a6206bffda9b","Type":"ContainerStarted","Data":"ca63844b5764040d608ba98ec5d419439e449f552b38531988746f546a34c3bd"} Jul 07 20:02:26.928516 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:26.928071 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2vmh4" event={"ID":"eb073572-f7a1-40e6-9485-a6206bffda9b","Type":"ContainerStarted","Data":"a2e38fc8d3305508b45799be97937e3ba22e5176e24888495b607b847c5411c7"} Jul 07 20:02:26.958516 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:26.958471 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-2vmh4" podStartSLOduration=4.106979688 podStartE2EDuration="5.958456376s" totalImagesPullingTime="1.851476688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:02:26.955550353 +0000 UTC m=+67.110012302" watchObservedRunningTime="2026-07-07 20:02:26.958456376 +0000 UTC m=+67.112918325" Jul 07 20:02:27.280327 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:27.280247 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da37b4bb-9b02-4491-9d87-c4f56be7c7ad-metrics-certs\") pod \"network-metrics-daemon-qvph6\" (UID: \"da37b4bb-9b02-4491-9d87-c4f56be7c7ad\") " pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:02:27.283608 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:27.283581 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 20:02:27.293336 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:27.293310 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da37b4bb-9b02-4491-9d87-c4f56be7c7ad-metrics-certs\") pod \"network-metrics-daemon-qvph6\" (UID: \"da37b4bb-9b02-4491-9d87-c4f56be7c7ad\") " pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:02:27.380675 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:27.380636 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17142129-e574-4092-bd8d-0ffa3de6507a-original-pull-secret\") pod \"global-pull-secret-syncer-zr7c2\" (UID: \"17142129-e574-4092-bd8d-0ffa3de6507a\") " pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:02:27.380857 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:27.380690 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spwbq\" (UniqueName: \"kubernetes.io/projected/92ea0d19-f83c-4b0d-969d-cf288f52c55a-kube-api-access-spwbq\") pod \"network-check-target-wj7w5\" (UID: \"92ea0d19-f83c-4b0d-969d-cf288f52c55a\") " pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:02:27.384084 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:27.384058 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 20:02:27.384596 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:27.384574 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 20:02:27.394159 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:27.394134 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17142129-e574-4092-bd8d-0ffa3de6507a-original-pull-secret\") pod \"global-pull-secret-syncer-zr7c2\" (UID: \"17142129-e574-4092-bd8d-0ffa3de6507a\") " pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:02:27.394263 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:27.394194 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 20:02:27.405080 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:27.405060 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spwbq\" (UniqueName: \"kubernetes.io/projected/92ea0d19-f83c-4b0d-969d-cf288f52c55a-kube-api-access-spwbq\") pod \"network-check-target-wj7w5\" (UID: \"92ea0d19-f83c-4b0d-969d-cf288f52c55a\") " pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:02:27.489595 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:27.489564 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr7c2" Jul 07 20:02:27.498586 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:27.498562 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lcjbq\"" Jul 07 20:02:27.505377 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:27.505186 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:02:27.505512 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:27.505432 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mcw2j\"" Jul 07 20:02:27.509636 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:27.509616 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvph6" Jul 07 20:02:27.683784 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:27.683716 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zr7c2"] Jul 07 20:02:27.686057 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:02:27.686020 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17142129_e574_4092_bd8d_0ffa3de6507a.slice/crio-e520943043dad379cd851f2e7e28bf1a59d7978de3a3a0431fc56e8b8ea84656 WatchSource:0}: Error finding container e520943043dad379cd851f2e7e28bf1a59d7978de3a3a0431fc56e8b8ea84656: Status 404 returned error can't find the container with id e520943043dad379cd851f2e7e28bf1a59d7978de3a3a0431fc56e8b8ea84656 Jul 07 20:02:27.695236 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:27.695215 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wj7w5"] Jul 07 20:02:27.698937 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:02:27.698912 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92ea0d19_f83c_4b0d_969d_cf288f52c55a.slice/crio-a6c077ee8413d273edf857bdcc3decf45a47f2d563547b33675ccd09bbb8eee3 WatchSource:0}: Error finding container a6c077ee8413d273edf857bdcc3decf45a47f2d563547b33675ccd09bbb8eee3: Status 404 returned error can't find the container with id a6c077ee8413d273edf857bdcc3decf45a47f2d563547b33675ccd09bbb8eee3 Jul 07 20:02:27.708939 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:27.708921 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qvph6"] Jul 07 20:02:27.710552 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:02:27.710529 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda37b4bb_9b02_4491_9d87_c4f56be7c7ad.slice/crio-dc2a4bfc526cbdf995db4e11a3ee74bc4ddabbd21b788f2b9a1ca3e31e40d56b WatchSource:0}: Error finding container dc2a4bfc526cbdf995db4e11a3ee74bc4ddabbd21b788f2b9a1ca3e31e40d56b: Status 404 returned error can't find the container with id dc2a4bfc526cbdf995db4e11a3ee74bc4ddabbd21b788f2b9a1ca3e31e40d56b Jul 07 20:02:27.932820 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:27.932776 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wj7w5" event={"ID":"92ea0d19-f83c-4b0d-969d-cf288f52c55a","Type":"ContainerStarted","Data":"a6c077ee8413d273edf857bdcc3decf45a47f2d563547b33675ccd09bbb8eee3"} Jul 07 20:02:27.934043 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:27.934011 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qvph6" event={"ID":"da37b4bb-9b02-4491-9d87-c4f56be7c7ad","Type":"ContainerStarted","Data":"dc2a4bfc526cbdf995db4e11a3ee74bc4ddabbd21b788f2b9a1ca3e31e40d56b"} Jul 07 20:02:27.935199 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:27.935169 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zr7c2" event={"ID":"17142129-e574-4092-bd8d-0ffa3de6507a","Type":"ContainerStarted","Data":"e520943043dad379cd851f2e7e28bf1a59d7978de3a3a0431fc56e8b8ea84656"} Jul 07 20:02:29.428367 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.428334 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-bsssh"] Jul 07 20:02:29.430755 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.430737 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-bsssh" Jul 07 20:02:29.436538 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.436511 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 07 20:02:29.436773 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.436746 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 07 20:02:29.437478 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.437453 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-ww4b9\"" Jul 07 20:02:29.449096 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.449077 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-bsdc6"] Jul 07 20:02:29.451174 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.451158 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-bsssh"] Jul 07 20:02:29.451263 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.451245 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-bsdc6" Jul 07 20:02:29.454718 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.454699 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 07 20:02:29.454864 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.454846 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 07 20:02:29.454964 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.454879 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wzzzs\"" Jul 07 20:02:29.455339 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.455323 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 07 20:02:29.465474 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.465455 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-bsdc6"] Jul 07 20:02:29.486342 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.486315 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-89snw"] Jul 07 20:02:29.488975 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.488950 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-89snw" Jul 07 20:02:29.492049 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.492026 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 07 20:02:29.492154 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.492055 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 07 20:02:29.492154 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.492078 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-24vrt\"" Jul 07 20:02:29.492271 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.492028 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 07 20:02:29.494654 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.494612 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cab569a8-8055-4e0f-bba9-c698e829a799-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-bsssh\" (UID: \"cab569a8-8055-4e0f-bba9-c698e829a799\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-bsssh" Jul 07 20:02:29.494769 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.494654 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1216713e-ac05-4a9b-94d0-5ec9e0caad6a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-bsdc6\" (UID: \"1216713e-ac05-4a9b-94d0-5ec9e0caad6a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-bsdc6" Jul 07 20:02:29.494769 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.494684 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1216713e-ac05-4a9b-94d0-5ec9e0caad6a-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-bsdc6\" (UID: \"1216713e-ac05-4a9b-94d0-5ec9e0caad6a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-bsdc6" Jul 07 20:02:29.494769 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.494735 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1216713e-ac05-4a9b-94d0-5ec9e0caad6a-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-bsdc6\" (UID: \"1216713e-ac05-4a9b-94d0-5ec9e0caad6a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-bsdc6" Jul 07 20:02:29.494936 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.494771 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cab569a8-8055-4e0f-bba9-c698e829a799-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-bsssh\" (UID: \"cab569a8-8055-4e0f-bba9-c698e829a799\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-bsssh" Jul 07 20:02:29.494936 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.494790 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2scqf\" (UniqueName: \"kubernetes.io/projected/cab569a8-8055-4e0f-bba9-c698e829a799-kube-api-access-2scqf\") pod \"openshift-state-metrics-54cf96dfcf-bsssh\" (UID: \"cab569a8-8055-4e0f-bba9-c698e829a799\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-bsssh" Jul 07 20:02:29.494936 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.494823 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cab569a8-8055-4e0f-bba9-c698e829a799-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-bsssh\" (UID: \"cab569a8-8055-4e0f-bba9-c698e829a799\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-bsssh" Jul 07 20:02:29.494936 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.494857 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1216713e-ac05-4a9b-94d0-5ec9e0caad6a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-bsdc6\" (UID: \"1216713e-ac05-4a9b-94d0-5ec9e0caad6a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-bsdc6" Jul 07 20:02:29.494936 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.494927 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzg8q\" (UniqueName: \"kubernetes.io/projected/1216713e-ac05-4a9b-94d0-5ec9e0caad6a-kube-api-access-zzg8q\") pod \"kube-state-metrics-7c9dbb8b7d-bsdc6\" (UID: \"1216713e-ac05-4a9b-94d0-5ec9e0caad6a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-bsdc6" Jul 07 20:02:29.495146 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.494951 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1216713e-ac05-4a9b-94d0-5ec9e0caad6a-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-bsdc6\" (UID: \"1216713e-ac05-4a9b-94d0-5ec9e0caad6a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-bsdc6" Jul 07 20:02:29.596180 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.596146 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1216713e-ac05-4a9b-94d0-5ec9e0caad6a-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-bsdc6\" (UID: \"1216713e-ac05-4a9b-94d0-5ec9e0caad6a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-bsdc6" Jul 07 20:02:29.596180 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.596186 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1216713e-ac05-4a9b-94d0-5ec9e0caad6a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-bsdc6\" (UID: \"1216713e-ac05-4a9b-94d0-5ec9e0caad6a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-bsdc6" Jul 07 20:02:29.596464 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.596205 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1216713e-ac05-4a9b-94d0-5ec9e0caad6a-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-bsdc6\" (UID: \"1216713e-ac05-4a9b-94d0-5ec9e0caad6a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-bsdc6" Jul 07 20:02:29.596464 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.596346 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18-node-exporter-tls\") pod \"node-exporter-89snw\" (UID: \"d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18\") " pod="openshift-monitoring/node-exporter-89snw" Jul 07 20:02:29.596464 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.596390 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18-node-exporter-accelerators-collector-config\") pod \"node-exporter-89snw\" (UID: \"d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18\") " pod="openshift-monitoring/node-exporter-89snw" Jul 07 20:02:29.596464 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.596444 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18-sys\") pod \"node-exporter-89snw\" (UID: \"d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18\") " pod="openshift-monitoring/node-exporter-89snw" Jul 07 20:02:29.596680 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.596509 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18-root\") pod \"node-exporter-89snw\" (UID: \"d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18\") " pod="openshift-monitoring/node-exporter-89snw" Jul 07 20:02:29.596680 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.596549 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18-node-exporter-wtmp\") pod \"node-exporter-89snw\" (UID: \"d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18\") " pod="openshift-monitoring/node-exporter-89snw" Jul 07 20:02:29.596680 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.596576 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18-metrics-client-ca\") pod \"node-exporter-89snw\" (UID: \"d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18\") " pod="openshift-monitoring/node-exporter-89snw" Jul 07 20:02:29.596680 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.596623 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzg8q\" (UniqueName: \"kubernetes.io/projected/1216713e-ac05-4a9b-94d0-5ec9e0caad6a-kube-api-access-zzg8q\") pod \"kube-state-metrics-7c9dbb8b7d-bsdc6\" (UID: \"1216713e-ac05-4a9b-94d0-5ec9e0caad6a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-bsdc6" Jul 07 20:02:29.596680 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.596654 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cab569a8-8055-4e0f-bba9-c698e829a799-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-bsssh\" (UID: \"cab569a8-8055-4e0f-bba9-c698e829a799\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-bsssh" Jul 07 20:02:29.596902 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.596684 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18-node-exporter-textfile\") pod \"node-exporter-89snw\" (UID: \"d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18\") " pod="openshift-monitoring/node-exporter-89snw" Jul 07 20:02:29.596902 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.596687 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1216713e-ac05-4a9b-94d0-5ec9e0caad6a-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-bsdc6\" (UID: \"1216713e-ac05-4a9b-94d0-5ec9e0caad6a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-bsdc6" Jul 07 20:02:29.596902 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.596714 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1216713e-ac05-4a9b-94d0-5ec9e0caad6a-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-bsdc6\" (UID: \"1216713e-ac05-4a9b-94d0-5ec9e0caad6a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-bsdc6" Jul 07 20:02:29.596902 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.596742 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cab569a8-8055-4e0f-bba9-c698e829a799-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-bsssh\" (UID: \"cab569a8-8055-4e0f-bba9-c698e829a799\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-bsssh" Jul 07 20:02:29.596902 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.596773 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2scqf\" (UniqueName: \"kubernetes.io/projected/cab569a8-8055-4e0f-bba9-c698e829a799-kube-api-access-2scqf\") pod \"openshift-state-metrics-54cf96dfcf-bsssh\" (UID: \"cab569a8-8055-4e0f-bba9-c698e829a799\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-bsssh" Jul 07 20:02:29.596902 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.596806 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cab569a8-8055-4e0f-bba9-c698e829a799-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-bsssh\" (UID: \"cab569a8-8055-4e0f-bba9-c698e829a799\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-bsssh" Jul 07 20:02:29.596902 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.596830 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-89snw\" (UID: \"d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18\") " pod="openshift-monitoring/node-exporter-89snw" Jul 07 20:02:29.596902 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.596859 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1216713e-ac05-4a9b-94d0-5ec9e0caad6a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-bsdc6\" (UID: \"1216713e-ac05-4a9b-94d0-5ec9e0caad6a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-bsdc6" Jul 07 20:02:29.596902 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.596902 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmw6n\" (UniqueName: \"kubernetes.io/projected/d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18-kube-api-access-qmw6n\") pod \"node-exporter-89snw\" (UID: \"d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18\") " pod="openshift-monitoring/node-exporter-89snw" Jul 07 20:02:29.597415 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:29.596971 2565 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 07 20:02:29.597415 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:29.597033 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cab569a8-8055-4e0f-bba9-c698e829a799-openshift-state-metrics-tls podName:cab569a8-8055-4e0f-bba9-c698e829a799 nodeName:}" failed. No retries permitted until 2026-07-07 20:02:30.097014033 +0000 UTC m=+70.251475964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/cab569a8-8055-4e0f-bba9-c698e829a799-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-bsssh" (UID: "cab569a8-8055-4e0f-bba9-c698e829a799") : secret "openshift-state-metrics-tls" not found Jul 07 20:02:29.597415 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.597079 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1216713e-ac05-4a9b-94d0-5ec9e0caad6a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-bsdc6\" (UID: \"1216713e-ac05-4a9b-94d0-5ec9e0caad6a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-bsdc6" Jul 07 20:02:29.597415 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.597173 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1216713e-ac05-4a9b-94d0-5ec9e0caad6a-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-bsdc6\" (UID: \"1216713e-ac05-4a9b-94d0-5ec9e0caad6a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-bsdc6" Jul 07 20:02:29.597645 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.597551 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cab569a8-8055-4e0f-bba9-c698e829a799-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-bsssh\" (UID: \"cab569a8-8055-4e0f-bba9-c698e829a799\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-bsssh" Jul 07 20:02:29.599907 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.599883 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1216713e-ac05-4a9b-94d0-5ec9e0caad6a-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-bsdc6\" (UID: \"1216713e-ac05-4a9b-94d0-5ec9e0caad6a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-bsdc6" Jul 07 20:02:29.600004 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.599923 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cab569a8-8055-4e0f-bba9-c698e829a799-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-bsssh\" (UID: \"cab569a8-8055-4e0f-bba9-c698e829a799\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-bsssh" Jul 07 20:02:29.600062 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.600026 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1216713e-ac05-4a9b-94d0-5ec9e0caad6a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-bsdc6\" (UID: \"1216713e-ac05-4a9b-94d0-5ec9e0caad6a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-bsdc6" Jul 07 20:02:29.611294 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.611266 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2scqf\" (UniqueName: \"kubernetes.io/projected/cab569a8-8055-4e0f-bba9-c698e829a799-kube-api-access-2scqf\") pod \"openshift-state-metrics-54cf96dfcf-bsssh\" (UID: \"cab569a8-8055-4e0f-bba9-c698e829a799\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-bsssh" Jul 07 20:02:29.612733 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.612702 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzg8q\" (UniqueName: \"kubernetes.io/projected/1216713e-ac05-4a9b-94d0-5ec9e0caad6a-kube-api-access-zzg8q\") pod \"kube-state-metrics-7c9dbb8b7d-bsdc6\" (UID: \"1216713e-ac05-4a9b-94d0-5ec9e0caad6a\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-bsdc6" Jul 07 20:02:29.698362 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.698294 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18-node-exporter-textfile\") pod \"node-exporter-89snw\" (UID: \"d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18\") " pod="openshift-monitoring/node-exporter-89snw" Jul 07 20:02:29.698362 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.698347 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-89snw\" (UID: \"d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18\") " pod="openshift-monitoring/node-exporter-89snw" Jul 07 20:02:29.698590 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.698375 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmw6n\" (UniqueName: \"kubernetes.io/projected/d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18-kube-api-access-qmw6n\") pod \"node-exporter-89snw\" (UID: \"d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18\") " pod="openshift-monitoring/node-exporter-89snw" Jul 07 20:02:29.698590 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.698439 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18-node-exporter-tls\") pod \"node-exporter-89snw\" (UID: \"d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18\") " pod="openshift-monitoring/node-exporter-89snw" Jul 07 20:02:29.698590 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.698462 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18-node-exporter-accelerators-collector-config\") pod \"node-exporter-89snw\" (UID: \"d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18\") " pod="openshift-monitoring/node-exporter-89snw" Jul 07 20:02:29.698590 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.698483 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18-sys\") pod \"node-exporter-89snw\" (UID: \"d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18\") " pod="openshift-monitoring/node-exporter-89snw" Jul 07 20:02:29.698784 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.698695 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18-node-exporter-textfile\") pod \"node-exporter-89snw\" (UID: \"d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18\") " pod="openshift-monitoring/node-exporter-89snw" Jul 07 20:02:29.698833 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.698775 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18-root\") pod \"node-exporter-89snw\" (UID: \"d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18\") " pod="openshift-monitoring/node-exporter-89snw" Jul 07 20:02:29.698833 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.698814 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18-node-exporter-wtmp\") pod \"node-exporter-89snw\" (UID: \"d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18\") " pod="openshift-monitoring/node-exporter-89snw" Jul 07 20:02:29.698918 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.698840 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18-metrics-client-ca\") pod \"node-exporter-89snw\" (UID: \"d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18\") " pod="openshift-monitoring/node-exporter-89snw" Jul 07 20:02:29.698918 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.698857 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18-root\") pod \"node-exporter-89snw\" (UID: \"d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18\") " pod="openshift-monitoring/node-exporter-89snw" Jul 07 20:02:29.698918 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.698840 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18-sys\") pod \"node-exporter-89snw\" (UID: \"d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18\") " pod="openshift-monitoring/node-exporter-89snw" Jul 07 20:02:29.699086 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.699016 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18-node-exporter-wtmp\") pod \"node-exporter-89snw\" (UID: \"d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18\") " pod="openshift-monitoring/node-exporter-89snw" Jul 07 20:02:29.699086 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.699061 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18-node-exporter-accelerators-collector-config\") pod \"node-exporter-89snw\" (UID: \"d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18\") " pod="openshift-monitoring/node-exporter-89snw" Jul 07 20:02:29.699500 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.699456 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18-metrics-client-ca\") pod \"node-exporter-89snw\" (UID: \"d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18\") " pod="openshift-monitoring/node-exporter-89snw" Jul 07 20:02:29.701072 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.701045 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18-node-exporter-tls\") pod \"node-exporter-89snw\" (UID: \"d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18\") " pod="openshift-monitoring/node-exporter-89snw" Jul 07 20:02:29.701597 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.701571 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-89snw\" (UID: \"d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18\") " pod="openshift-monitoring/node-exporter-89snw" Jul 07 20:02:29.709107 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.709083 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmw6n\" (UniqueName: \"kubernetes.io/projected/d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18-kube-api-access-qmw6n\") pod \"node-exporter-89snw\" (UID: \"d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18\") " pod="openshift-monitoring/node-exporter-89snw" Jul 07 20:02:29.761448 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.761221 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-bsdc6" Jul 07 20:02:29.799967 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.799928 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-89snw" Jul 07 20:02:29.946155 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.946082 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2hcpv" event={"ID":"bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a","Type":"ContainerStarted","Data":"5d9669437b386f2c403aa5ace40e8b7da2922b39e165761a9373c12c02e1a921"} Jul 07 20:02:29.953674 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:02:29.953636 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd57ed5e1_7eb2_41c1_bcf7_edcd986e2a18.slice/crio-f1ace66160f9a52723fef1eb270b812b8425f3b417b6fdfd2ba957fcb59a160a WatchSource:0}: Error finding container f1ace66160f9a52723fef1eb270b812b8425f3b417b6fdfd2ba957fcb59a160a: Status 404 returned error can't find the container with id f1ace66160f9a52723fef1eb270b812b8425f3b417b6fdfd2ba957fcb59a160a Jul 07 20:02:29.971979 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:29.971916 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2hcpv" podStartSLOduration=2.247451537 podStartE2EDuration="11.971895059s" totalImagesPullingTime="9.724443522s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 20:02:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:02:29.970229011 +0000 UTC m=+70.124690960" watchObservedRunningTime="2026-07-07 20:02:29.971895059 +0000 UTC m=+70.126357009" Jul 07 20:02:30.105744 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.104120 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cab569a8-8055-4e0f-bba9-c698e829a799-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-bsssh\" (UID: \"cab569a8-8055-4e0f-bba9-c698e829a799\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-bsssh" Jul 07 20:02:30.105744 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:30.104320 2565 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 07 20:02:30.105744 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:02:30.104383 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cab569a8-8055-4e0f-bba9-c698e829a799-openshift-state-metrics-tls podName:cab569a8-8055-4e0f-bba9-c698e829a799 nodeName:}" failed. No retries permitted until 2026-07-07 20:02:31.104357062 +0000 UTC m=+71.258819007 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/cab569a8-8055-4e0f-bba9-c698e829a799-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-bsssh" (UID: "cab569a8-8055-4e0f-bba9-c698e829a799") : secret "openshift-state-metrics-tls" not found Jul 07 20:02:30.109814 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.109787 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-bsdc6"] Jul 07 20:02:30.113005 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:02:30.112981 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1216713e_ac05_4a9b_94d0_5ec9e0caad6a.slice/crio-60f00d0f25416166bd29339cf55b4d6120fa724914ac4c4fd0fc03c53c8d6d86 WatchSource:0}: Error finding container 60f00d0f25416166bd29339cf55b4d6120fa724914ac4c4fd0fc03c53c8d6d86: Status 404 returned error can't find the container with id 60f00d0f25416166bd29339cf55b4d6120fa724914ac4c4fd0fc03c53c8d6d86 Jul 07 20:02:30.526375 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.526335 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 20:02:30.529881 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.529836 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.534445 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.534420 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 07 20:02:30.534445 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.534434 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 07 20:02:30.534644 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.534431 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 07 20:02:30.535044 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.534999 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 07 20:02:30.535288 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.535198 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 07 20:02:30.536254 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.535427 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 07 20:02:30.536254 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.535586 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hqvjf\"" Jul 07 20:02:30.536254 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.535681 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 07 20:02:30.536254 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.535816 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 07 20:02:30.536254 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.536004 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 07 20:02:30.549179 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.549148 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 20:02:30.609164 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.609128 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f57d600-9387-4aa5-8798-1d7119506175-config-out\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.609333 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.609183 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.609333 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.609220 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-web-config\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.609333 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.609303 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f57d600-9387-4aa5-8798-1d7119506175-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.609516 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.609341 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmxhc\" (UniqueName: \"kubernetes.io/projected/1f57d600-9387-4aa5-8798-1d7119506175-kube-api-access-tmxhc\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.609516 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.609383 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.609516 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.609428 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f57d600-9387-4aa5-8798-1d7119506175-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.609516 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.609454 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f57d600-9387-4aa5-8798-1d7119506175-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.609516 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.609487 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.609754 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.609521 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-config-volume\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.609754 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.609552 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.609754 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.609578 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1f57d600-9387-4aa5-8798-1d7119506175-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.609754 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.609607 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.710060 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.710023 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f57d600-9387-4aa5-8798-1d7119506175-config-out\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.710219 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.710078 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.710219 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.710112 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-web-config\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.710219 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.710145 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f57d600-9387-4aa5-8798-1d7119506175-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.710219 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.710170 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmxhc\" (UniqueName: \"kubernetes.io/projected/1f57d600-9387-4aa5-8798-1d7119506175-kube-api-access-tmxhc\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.710219 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.710199 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.710219 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.710216 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f57d600-9387-4aa5-8798-1d7119506175-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.710539 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.710233 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f57d600-9387-4aa5-8798-1d7119506175-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.710539 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.710251 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.710539 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.710273 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-config-volume\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.710539 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.710291 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.710539 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.710309 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1f57d600-9387-4aa5-8798-1d7119506175-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.710539 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.710330 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.711346 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.711014 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f57d600-9387-4aa5-8798-1d7119506175-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.711723 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.711695 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f57d600-9387-4aa5-8798-1d7119506175-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.711997 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.711973 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1f57d600-9387-4aa5-8798-1d7119506175-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.714521 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.714267 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.714521 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.714473 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f57d600-9387-4aa5-8798-1d7119506175-config-out\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.714685 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.714579 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.714685 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.714605 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.714987 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.714945 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-config-volume\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.714987 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.714956 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.716108 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.716083 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-web-config\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.716204 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.716135 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.716261 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.716243 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f57d600-9387-4aa5-8798-1d7119506175-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.720112 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.720089 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmxhc\" (UniqueName: \"kubernetes.io/projected/1f57d600-9387-4aa5-8798-1d7119506175-kube-api-access-tmxhc\") pod \"alertmanager-main-0\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.842286 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.842204 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:02:30.952263 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.952225 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-bsdc6" event={"ID":"1216713e-ac05-4a9b-94d0-5ec9e0caad6a","Type":"ContainerStarted","Data":"60f00d0f25416166bd29339cf55b4d6120fa724914ac4c4fd0fc03c53c8d6d86"} Jul 07 20:02:30.953905 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.953702 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-89snw" event={"ID":"d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18","Type":"ContainerStarted","Data":"f1ace66160f9a52723fef1eb270b812b8425f3b417b6fdfd2ba957fcb59a160a"} Jul 07 20:02:30.956956 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.956804 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qvph6" event={"ID":"da37b4bb-9b02-4491-9d87-c4f56be7c7ad","Type":"ContainerStarted","Data":"2095b8db77dc0180c072ad49da1bd53e51537d25b8910268a2fe9dd62780606e"} Jul 07 20:02:30.956956 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.956864 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qvph6" event={"ID":"da37b4bb-9b02-4491-9d87-c4f56be7c7ad","Type":"ContainerStarted","Data":"78deb441c319054b3c344e626d3f8f86ca83994e03d48efda2f5626ffe0759ad"} Jul 07 20:02:30.977203 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:30.977059 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qvph6" podStartSLOduration=67.708756114 podStartE2EDuration="1m9.977039333s" totalImagesPullingTime="2.268283219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:01:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:02:30.976221868 +0000 UTC m=+71.130683818" watchObservedRunningTime="2026-07-07 20:02:30.977039333 +0000 UTC m=+71.131501283" Jul 07 20:02:31.114424 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.114297 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cab569a8-8055-4e0f-bba9-c698e829a799-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-bsssh\" (UID: \"cab569a8-8055-4e0f-bba9-c698e829a799\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-bsssh" Jul 07 20:02:31.118142 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.118088 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cab569a8-8055-4e0f-bba9-c698e829a799-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-bsssh\" (UID: \"cab569a8-8055-4e0f-bba9-c698e829a799\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-bsssh" Jul 07 20:02:31.173926 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.173892 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 20:02:31.178242 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:02:31.178214 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f57d600_9387_4aa5_8798_1d7119506175.slice/crio-49f31fe07412ced946aa6c14d8faafcd584d529c94a8bf92e7f80fffad3c4727 WatchSource:0}: Error finding container 49f31fe07412ced946aa6c14d8faafcd584d529c94a8bf92e7f80fffad3c4727: Status 404 returned error can't find the container with id 49f31fe07412ced946aa6c14d8faafcd584d529c94a8bf92e7f80fffad3c4727 Jul 07 20:02:31.239452 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.239420 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-bsssh" Jul 07 20:02:31.401367 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.401274 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-bsssh"] Jul 07 20:02:31.404773 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:02:31.404746 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab569a8_8055_4e0f_bba9_c698e829a799.slice/crio-9117ff54d9ea05ece002835ec98bd3f0254435a047b79b4df23a8f9127310a16 WatchSource:0}: Error finding container 9117ff54d9ea05ece002835ec98bd3f0254435a047b79b4df23a8f9127310a16: Status 404 returned error can't find the container with id 9117ff54d9ea05ece002835ec98bd3f0254435a047b79b4df23a8f9127310a16 Jul 07 20:02:31.544451 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.544423 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-677c97cfd9-8jl49"] Jul 07 20:02:31.547362 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.547340 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" Jul 07 20:02:31.551252 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.551226 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 07 20:02:31.561738 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.561371 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 07 20:02:31.566086 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.566067 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 07 20:02:31.566357 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.566226 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ag27ulvpdqknj\"" Jul 07 20:02:31.566357 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.566235 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 07 20:02:31.566632 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.566607 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 07 20:02:31.570762 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.570742 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-mdn82\"" Jul 07 20:02:31.608032 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.608002 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-677c97cfd9-8jl49"] Jul 07 20:02:31.619105 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.619013 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1a5cb3fd-134b-4b74-b687-8b05bdeb31b6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-677c97cfd9-8jl49\" (UID: \"1a5cb3fd-134b-4b74-b687-8b05bdeb31b6\") " pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" Jul 07 20:02:31.619105 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.619058 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1a5cb3fd-134b-4b74-b687-8b05bdeb31b6-secret-grpc-tls\") pod \"thanos-querier-677c97cfd9-8jl49\" (UID: \"1a5cb3fd-134b-4b74-b687-8b05bdeb31b6\") " pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" Jul 07 20:02:31.619323 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.619137 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1a5cb3fd-134b-4b74-b687-8b05bdeb31b6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-677c97cfd9-8jl49\" (UID: \"1a5cb3fd-134b-4b74-b687-8b05bdeb31b6\") " pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" Jul 07 20:02:31.619323 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.619204 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9lss\" (UniqueName: \"kubernetes.io/projected/1a5cb3fd-134b-4b74-b687-8b05bdeb31b6-kube-api-access-h9lss\") pod \"thanos-querier-677c97cfd9-8jl49\" (UID: \"1a5cb3fd-134b-4b74-b687-8b05bdeb31b6\") " pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" Jul 07 20:02:31.619323 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.619250 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a5cb3fd-134b-4b74-b687-8b05bdeb31b6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-677c97cfd9-8jl49\" (UID: \"1a5cb3fd-134b-4b74-b687-8b05bdeb31b6\") " pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" Jul 07 20:02:31.619323 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.619301 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a5cb3fd-134b-4b74-b687-8b05bdeb31b6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-677c97cfd9-8jl49\" (UID: \"1a5cb3fd-134b-4b74-b687-8b05bdeb31b6\") " pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" Jul 07 20:02:31.619547 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.619325 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a5cb3fd-134b-4b74-b687-8b05bdeb31b6-metrics-client-ca\") pod \"thanos-querier-677c97cfd9-8jl49\" (UID: \"1a5cb3fd-134b-4b74-b687-8b05bdeb31b6\") " pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" Jul 07 20:02:31.619547 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.619365 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1a5cb3fd-134b-4b74-b687-8b05bdeb31b6-secret-thanos-querier-tls\") pod \"thanos-querier-677c97cfd9-8jl49\" (UID: \"1a5cb3fd-134b-4b74-b687-8b05bdeb31b6\") " pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" Jul 07 20:02:31.719834 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.719748 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1a5cb3fd-134b-4b74-b687-8b05bdeb31b6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-677c97cfd9-8jl49\" (UID: \"1a5cb3fd-134b-4b74-b687-8b05bdeb31b6\") " pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" Jul 07 20:02:31.719834 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.719789 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1a5cb3fd-134b-4b74-b687-8b05bdeb31b6-secret-grpc-tls\") pod \"thanos-querier-677c97cfd9-8jl49\" (UID: \"1a5cb3fd-134b-4b74-b687-8b05bdeb31b6\") " pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" Jul 07 20:02:31.719834 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.719818 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1a5cb3fd-134b-4b74-b687-8b05bdeb31b6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-677c97cfd9-8jl49\" (UID: \"1a5cb3fd-134b-4b74-b687-8b05bdeb31b6\") " pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" Jul 07 20:02:31.720097 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.719843 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9lss\" (UniqueName: \"kubernetes.io/projected/1a5cb3fd-134b-4b74-b687-8b05bdeb31b6-kube-api-access-h9lss\") pod \"thanos-querier-677c97cfd9-8jl49\" (UID: \"1a5cb3fd-134b-4b74-b687-8b05bdeb31b6\") " pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" Jul 07 20:02:31.720097 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.719875 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a5cb3fd-134b-4b74-b687-8b05bdeb31b6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-677c97cfd9-8jl49\" (UID: \"1a5cb3fd-134b-4b74-b687-8b05bdeb31b6\") " pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" Jul 07 20:02:31.720199 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.720135 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a5cb3fd-134b-4b74-b687-8b05bdeb31b6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-677c97cfd9-8jl49\" (UID: \"1a5cb3fd-134b-4b74-b687-8b05bdeb31b6\") " pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" Jul 07 20:02:31.720492 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.720469 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a5cb3fd-134b-4b74-b687-8b05bdeb31b6-metrics-client-ca\") pod \"thanos-querier-677c97cfd9-8jl49\" (UID: \"1a5cb3fd-134b-4b74-b687-8b05bdeb31b6\") " pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" Jul 07 20:02:31.720587 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.720528 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1a5cb3fd-134b-4b74-b687-8b05bdeb31b6-secret-thanos-querier-tls\") pod \"thanos-querier-677c97cfd9-8jl49\" (UID: \"1a5cb3fd-134b-4b74-b687-8b05bdeb31b6\") " pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" Jul 07 20:02:31.721241 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.721190 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a5cb3fd-134b-4b74-b687-8b05bdeb31b6-metrics-client-ca\") pod \"thanos-querier-677c97cfd9-8jl49\" (UID: \"1a5cb3fd-134b-4b74-b687-8b05bdeb31b6\") " pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" Jul 07 20:02:31.722871 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.722845 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a5cb3fd-134b-4b74-b687-8b05bdeb31b6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-677c97cfd9-8jl49\" (UID: \"1a5cb3fd-134b-4b74-b687-8b05bdeb31b6\") " pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" Jul 07 20:02:31.722959 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.722908 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/1a5cb3fd-134b-4b74-b687-8b05bdeb31b6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-677c97cfd9-8jl49\" (UID: \"1a5cb3fd-134b-4b74-b687-8b05bdeb31b6\") " pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" Jul 07 20:02:31.723112 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.723088 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a5cb3fd-134b-4b74-b687-8b05bdeb31b6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-677c97cfd9-8jl49\" (UID: \"1a5cb3fd-134b-4b74-b687-8b05bdeb31b6\") " pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" Jul 07 20:02:31.723552 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.723512 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/1a5cb3fd-134b-4b74-b687-8b05bdeb31b6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-677c97cfd9-8jl49\" (UID: \"1a5cb3fd-134b-4b74-b687-8b05bdeb31b6\") " pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" Jul 07 20:02:31.723887 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.723862 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/1a5cb3fd-134b-4b74-b687-8b05bdeb31b6-secret-thanos-querier-tls\") pod \"thanos-querier-677c97cfd9-8jl49\" (UID: \"1a5cb3fd-134b-4b74-b687-8b05bdeb31b6\") " pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" Jul 07 20:02:31.724040 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.724018 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1a5cb3fd-134b-4b74-b687-8b05bdeb31b6-secret-grpc-tls\") pod \"thanos-querier-677c97cfd9-8jl49\" (UID: \"1a5cb3fd-134b-4b74-b687-8b05bdeb31b6\") " pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" Jul 07 20:02:31.730027 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.730004 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9lss\" (UniqueName: \"kubernetes.io/projected/1a5cb3fd-134b-4b74-b687-8b05bdeb31b6-kube-api-access-h9lss\") pod \"thanos-querier-677c97cfd9-8jl49\" (UID: \"1a5cb3fd-134b-4b74-b687-8b05bdeb31b6\") " pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" Jul 07 20:02:31.867029 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.866992 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" Jul 07 20:02:31.962842 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.962801 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f57d600-9387-4aa5-8798-1d7119506175","Type":"ContainerStarted","Data":"49f31fe07412ced946aa6c14d8faafcd584d529c94a8bf92e7f80fffad3c4727"} Jul 07 20:02:31.966465 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.966295 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-bsssh" event={"ID":"cab569a8-8055-4e0f-bba9-c698e829a799","Type":"ContainerStarted","Data":"a8fb28f41a5b61d24ecb033722b7acac2ac9256a392856969e349edd9623a619"} Jul 07 20:02:31.966465 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.966377 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-bsssh" event={"ID":"cab569a8-8055-4e0f-bba9-c698e829a799","Type":"ContainerStarted","Data":"80fdddac5eb903770738e3233dfd49ba4934bae358d2f255520806aaaf6dc44b"} Jul 07 20:02:31.966465 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.966428 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-bsssh" event={"ID":"cab569a8-8055-4e0f-bba9-c698e829a799","Type":"ContainerStarted","Data":"9117ff54d9ea05ece002835ec98bd3f0254435a047b79b4df23a8f9127310a16"} Jul 07 20:02:31.968264 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.968231 2565 generic.go:356] "Generic (PLEG): container finished" podID="d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18" containerID="7a761fc0d00e704fbcbbe5666ea574a578723684fd22e7d7b538d17fb8bdbd13" exitCode=0 Jul 07 20:02:31.968377 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:31.968319 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-89snw" event={"ID":"d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18","Type":"ContainerDied","Data":"7a761fc0d00e704fbcbbe5666ea574a578723684fd22e7d7b538d17fb8bdbd13"} Jul 07 20:02:32.028118 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:32.028083 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-677c97cfd9-8jl49"] Jul 07 20:02:32.974001 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:32.973965 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-89snw" event={"ID":"d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18","Type":"ContainerStarted","Data":"765817fd8c37e245d21d7b4c44b0671c7b3cfd0f9a5a04e9726d99f580b338bf"} Jul 07 20:02:33.478047 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:02:33.478021 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a5cb3fd_134b_4b74_b687_8b05bdeb31b6.slice/crio-1cfe66236ee2218279dbe7c516bc25c680da381f53e2d23cd87a40fcb78fcd66 WatchSource:0}: Error finding container 1cfe66236ee2218279dbe7c516bc25c680da381f53e2d23cd87a40fcb78fcd66: Status 404 returned error can't find the container with id 1cfe66236ee2218279dbe7c516bc25c680da381f53e2d23cd87a40fcb78fcd66 Jul 07 20:02:33.815194 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:33.815112 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-76d99668b4-2x4pv"] Jul 07 20:02:33.831155 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:33.831128 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-76d99668b4-2x4pv" Jul 07 20:02:33.842674 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:33.842649 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-76d99668b4-2x4pv"] Jul 07 20:02:33.842900 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:33.842697 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 07 20:02:33.843819 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:33.843462 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 07 20:02:33.843819 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:33.843649 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 07 20:02:33.845711 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:33.844147 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-e87u4a7lmo687\"" Jul 07 20:02:33.845711 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:33.844345 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-xtmtt\"" Jul 07 20:02:33.845711 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:33.845115 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 07 20:02:33.940035 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:33.939946 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6eede5f7-90c3-4d64-a6bc-ff8580fb3a33-secret-metrics-server-tls\") pod \"metrics-server-76d99668b4-2x4pv\" (UID: \"6eede5f7-90c3-4d64-a6bc-ff8580fb3a33\") " pod="openshift-monitoring/metrics-server-76d99668b4-2x4pv" Jul 07 20:02:33.940196 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:33.940050 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6eede5f7-90c3-4d64-a6bc-ff8580fb3a33-metrics-server-audit-profiles\") pod \"metrics-server-76d99668b4-2x4pv\" (UID: \"6eede5f7-90c3-4d64-a6bc-ff8580fb3a33\") " pod="openshift-monitoring/metrics-server-76d99668b4-2x4pv" Jul 07 20:02:33.940196 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:33.940129 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6eede5f7-90c3-4d64-a6bc-ff8580fb3a33-secret-metrics-server-client-certs\") pod \"metrics-server-76d99668b4-2x4pv\" (UID: \"6eede5f7-90c3-4d64-a6bc-ff8580fb3a33\") " pod="openshift-monitoring/metrics-server-76d99668b4-2x4pv" Jul 07 20:02:33.940196 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:33.940152 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6eede5f7-90c3-4d64-a6bc-ff8580fb3a33-audit-log\") pod \"metrics-server-76d99668b4-2x4pv\" (UID: \"6eede5f7-90c3-4d64-a6bc-ff8580fb3a33\") " pod="openshift-monitoring/metrics-server-76d99668b4-2x4pv" Jul 07 20:02:33.940196 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:33.940180 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6eede5f7-90c3-4d64-a6bc-ff8580fb3a33-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76d99668b4-2x4pv\" (UID: \"6eede5f7-90c3-4d64-a6bc-ff8580fb3a33\") " pod="openshift-monitoring/metrics-server-76d99668b4-2x4pv" Jul 07 20:02:33.940464 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:33.940243 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5jdp\" (UniqueName: \"kubernetes.io/projected/6eede5f7-90c3-4d64-a6bc-ff8580fb3a33-kube-api-access-j5jdp\") pod \"metrics-server-76d99668b4-2x4pv\" (UID: \"6eede5f7-90c3-4d64-a6bc-ff8580fb3a33\") " pod="openshift-monitoring/metrics-server-76d99668b4-2x4pv" Jul 07 20:02:33.940464 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:33.940312 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6eede5f7-90c3-4d64-a6bc-ff8580fb3a33-client-ca-bundle\") pod \"metrics-server-76d99668b4-2x4pv\" (UID: \"6eede5f7-90c3-4d64-a6bc-ff8580fb3a33\") " pod="openshift-monitoring/metrics-server-76d99668b4-2x4pv" Jul 07 20:02:33.980448 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:33.980413 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" event={"ID":"1a5cb3fd-134b-4b74-b687-8b05bdeb31b6","Type":"ContainerStarted","Data":"1cfe66236ee2218279dbe7c516bc25c680da381f53e2d23cd87a40fcb78fcd66"} Jul 07 20:02:34.041035 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:34.040999 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6eede5f7-90c3-4d64-a6bc-ff8580fb3a33-secret-metrics-server-client-certs\") pod \"metrics-server-76d99668b4-2x4pv\" (UID: \"6eede5f7-90c3-4d64-a6bc-ff8580fb3a33\") " pod="openshift-monitoring/metrics-server-76d99668b4-2x4pv" Jul 07 20:02:34.041192 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:34.041048 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6eede5f7-90c3-4d64-a6bc-ff8580fb3a33-audit-log\") pod \"metrics-server-76d99668b4-2x4pv\" (UID: \"6eede5f7-90c3-4d64-a6bc-ff8580fb3a33\") " pod="openshift-monitoring/metrics-server-76d99668b4-2x4pv" Jul 07 20:02:34.041192 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:34.041086 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6eede5f7-90c3-4d64-a6bc-ff8580fb3a33-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76d99668b4-2x4pv\" (UID: \"6eede5f7-90c3-4d64-a6bc-ff8580fb3a33\") " pod="openshift-monitoring/metrics-server-76d99668b4-2x4pv" Jul 07 20:02:34.041192 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:34.041104 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5jdp\" (UniqueName: \"kubernetes.io/projected/6eede5f7-90c3-4d64-a6bc-ff8580fb3a33-kube-api-access-j5jdp\") pod \"metrics-server-76d99668b4-2x4pv\" (UID: \"6eede5f7-90c3-4d64-a6bc-ff8580fb3a33\") " pod="openshift-monitoring/metrics-server-76d99668b4-2x4pv" Jul 07 20:02:34.041192 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:34.041156 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6eede5f7-90c3-4d64-a6bc-ff8580fb3a33-client-ca-bundle\") pod \"metrics-server-76d99668b4-2x4pv\" (UID: \"6eede5f7-90c3-4d64-a6bc-ff8580fb3a33\") " pod="openshift-monitoring/metrics-server-76d99668b4-2x4pv" Jul 07 20:02:34.041428 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:34.041385 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6eede5f7-90c3-4d64-a6bc-ff8580fb3a33-secret-metrics-server-tls\") pod \"metrics-server-76d99668b4-2x4pv\" (UID: \"6eede5f7-90c3-4d64-a6bc-ff8580fb3a33\") " pod="openshift-monitoring/metrics-server-76d99668b4-2x4pv" Jul 07 20:02:34.041500 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:34.041458 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6eede5f7-90c3-4d64-a6bc-ff8580fb3a33-metrics-server-audit-profiles\") pod \"metrics-server-76d99668b4-2x4pv\" (UID: \"6eede5f7-90c3-4d64-a6bc-ff8580fb3a33\") " pod="openshift-monitoring/metrics-server-76d99668b4-2x4pv" Jul 07 20:02:34.041574 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:34.041545 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6eede5f7-90c3-4d64-a6bc-ff8580fb3a33-audit-log\") pod \"metrics-server-76d99668b4-2x4pv\" (UID: \"6eede5f7-90c3-4d64-a6bc-ff8580fb3a33\") " pod="openshift-monitoring/metrics-server-76d99668b4-2x4pv" Jul 07 20:02:34.042091 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:34.042022 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6eede5f7-90c3-4d64-a6bc-ff8580fb3a33-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76d99668b4-2x4pv\" (UID: \"6eede5f7-90c3-4d64-a6bc-ff8580fb3a33\") " pod="openshift-monitoring/metrics-server-76d99668b4-2x4pv" Jul 07 20:02:34.042484 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:34.042462 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6eede5f7-90c3-4d64-a6bc-ff8580fb3a33-metrics-server-audit-profiles\") pod \"metrics-server-76d99668b4-2x4pv\" (UID: \"6eede5f7-90c3-4d64-a6bc-ff8580fb3a33\") " pod="openshift-monitoring/metrics-server-76d99668b4-2x4pv" Jul 07 20:02:34.044183 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:34.044162 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6eede5f7-90c3-4d64-a6bc-ff8580fb3a33-secret-metrics-server-client-certs\") pod \"metrics-server-76d99668b4-2x4pv\" (UID: \"6eede5f7-90c3-4d64-a6bc-ff8580fb3a33\") " pod="openshift-monitoring/metrics-server-76d99668b4-2x4pv" Jul 07 20:02:34.044278 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:34.044181 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6eede5f7-90c3-4d64-a6bc-ff8580fb3a33-secret-metrics-server-tls\") pod \"metrics-server-76d99668b4-2x4pv\" (UID: \"6eede5f7-90c3-4d64-a6bc-ff8580fb3a33\") " pod="openshift-monitoring/metrics-server-76d99668b4-2x4pv" Jul 07 20:02:34.044316 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:34.044278 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6eede5f7-90c3-4d64-a6bc-ff8580fb3a33-client-ca-bundle\") pod \"metrics-server-76d99668b4-2x4pv\" (UID: \"6eede5f7-90c3-4d64-a6bc-ff8580fb3a33\") " pod="openshift-monitoring/metrics-server-76d99668b4-2x4pv" Jul 07 20:02:34.053601 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:34.053577 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5jdp\" (UniqueName: \"kubernetes.io/projected/6eede5f7-90c3-4d64-a6bc-ff8580fb3a33-kube-api-access-j5jdp\") pod \"metrics-server-76d99668b4-2x4pv\" (UID: \"6eede5f7-90c3-4d64-a6bc-ff8580fb3a33\") " pod="openshift-monitoring/metrics-server-76d99668b4-2x4pv" Jul 07 20:02:34.142476 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:34.142441 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-76d99668b4-2x4pv" Jul 07 20:02:34.152826 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:34.152796 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-nd79v"] Jul 07 20:02:34.164382 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:34.164364 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-nd79v" Jul 07 20:02:34.169658 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:34.169523 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-nd79v"] Jul 07 20:02:34.171370 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:34.170585 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-wsgqg\"" Jul 07 20:02:34.171370 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:34.170587 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 07 20:02:34.243281 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:34.243246 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7301ae59-b24d-4bda-a5c0-7b8036966f8e-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-nd79v\" (UID: \"7301ae59-b24d-4bda-a5c0-7b8036966f8e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-nd79v" Jul 07 20:02:34.243448 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:34.243295 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q6vv\" (UniqueName: \"kubernetes.io/projected/7301ae59-b24d-4bda-a5c0-7b8036966f8e-kube-api-access-9q6vv\") pod \"monitoring-plugin-d5dcff7cf-nd79v\" (UID: \"7301ae59-b24d-4bda-a5c0-7b8036966f8e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-nd79v" Jul 07 20:02:34.344246 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:34.344212 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7301ae59-b24d-4bda-a5c0-7b8036966f8e-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-nd79v\" (UID: \"7301ae59-b24d-4bda-a5c0-7b8036966f8e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-nd79v" Jul 07 20:02:34.344246 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:34.344249 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q6vv\" (UniqueName: \"kubernetes.io/projected/7301ae59-b24d-4bda-a5c0-7b8036966f8e-kube-api-access-9q6vv\") pod \"monitoring-plugin-d5dcff7cf-nd79v\" (UID: \"7301ae59-b24d-4bda-a5c0-7b8036966f8e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-nd79v" Jul 07 20:02:34.347144 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:34.347109 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7301ae59-b24d-4bda-a5c0-7b8036966f8e-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-nd79v\" (UID: \"7301ae59-b24d-4bda-a5c0-7b8036966f8e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-nd79v" Jul 07 20:02:34.360036 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:34.360005 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q6vv\" (UniqueName: \"kubernetes.io/projected/7301ae59-b24d-4bda-a5c0-7b8036966f8e-kube-api-access-9q6vv\") pod \"monitoring-plugin-d5dcff7cf-nd79v\" (UID: \"7301ae59-b24d-4bda-a5c0-7b8036966f8e\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-nd79v" Jul 07 20:02:34.478201 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:34.478112 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-nd79v" Jul 07 20:02:34.924314 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:34.924287 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4wwvr" Jul 07 20:02:35.816142 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.815822 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 20:02:35.820762 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.820736 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:35.828797 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.828766 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 07 20:02:35.829088 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.829069 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 07 20:02:35.829246 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.829214 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 07 20:02:35.829666 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.829650 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 07 20:02:35.830095 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.829924 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 07 20:02:35.830095 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.829959 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 07 20:02:35.830095 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.830049 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 07 20:02:35.830095 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.830060 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 07 20:02:35.830704 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.830195 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 07 20:02:35.830704 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.830293 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 07 20:02:35.830704 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.830486 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 07 20:02:35.830704 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.830564 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 07 20:02:35.830704 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.830615 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fpfzr\"" Jul 07 20:02:35.831358 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.831338 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 07 20:02:35.831484 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.831425 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9f8o4jglddmpe\"" Jul 07 20:02:35.833351 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.833173 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 07 20:02:35.842507 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.842473 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 20:02:35.962524 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.961678 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:35.962524 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.961723 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:35.962524 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.961762 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:35.962524 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.961785 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bb45973a-992c-47af-ba0b-201f02b6a702-config-out\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:35.962524 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.961809 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:35.962524 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.961831 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bb45973a-992c-47af-ba0b-201f02b6a702-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:35.962524 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.961864 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-web-config\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:35.962524 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.961906 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-config\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:35.962524 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.961929 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhn7j\" (UniqueName: \"kubernetes.io/projected/bb45973a-992c-47af-ba0b-201f02b6a702-kube-api-access-xhn7j\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:35.962524 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.961961 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bb45973a-992c-47af-ba0b-201f02b6a702-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:35.962524 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.962011 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:35.962524 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.962043 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:35.962524 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.962066 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:35.962524 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.962096 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:35.962524 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.962126 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:35.962524 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.962148 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:35.963575 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.962183 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:35.963575 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.962208 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:35.963575 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.962247 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:35.963575 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:35.962295 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.009521 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.006607 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-89snw" event={"ID":"d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18","Type":"ContainerStarted","Data":"e2f588f5fd8894f6c0deb867a5777289b501d7543f5f93db6b0dbaf51ccaa483"} Jul 07 20:02:36.064246 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.063534 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.064246 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.063576 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.064246 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.063608 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.064246 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.063634 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.064246 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.063666 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.064246 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.063703 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.064246 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.063729 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.064246 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.063751 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.064246 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.063780 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.064246 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.063806 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bb45973a-992c-47af-ba0b-201f02b6a702-config-out\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.064246 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.063829 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.064246 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.063851 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bb45973a-992c-47af-ba0b-201f02b6a702-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.064246 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.063883 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-web-config\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.064246 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.063923 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-config\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.064246 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.063944 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhn7j\" (UniqueName: \"kubernetes.io/projected/bb45973a-992c-47af-ba0b-201f02b6a702-kube-api-access-xhn7j\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.064246 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.063975 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bb45973a-992c-47af-ba0b-201f02b6a702-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.064246 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.064025 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.065203 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.064058 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.065203 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.064083 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.072648 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.069138 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.072648 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.070041 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.072648 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.071803 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.072648 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.071839 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.072648 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.072150 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bb45973a-992c-47af-ba0b-201f02b6a702-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.076901 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.075344 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.076901 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.075389 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.076901 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.076252 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.076901 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.076499 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.076901 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.076697 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.077256 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.077197 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.085735 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.084290 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-web-config\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.085934 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.085687 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.086169 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.086124 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-config\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.090542 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.087527 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.090542 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.088056 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhn7j\" (UniqueName: \"kubernetes.io/projected/bb45973a-992c-47af-ba0b-201f02b6a702-kube-api-access-xhn7j\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.090542 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.089341 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bb45973a-992c-47af-ba0b-201f02b6a702-config-out\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.090542 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.089874 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.090542 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.090324 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.090542 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.090326 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bb45973a-992c-47af-ba0b-201f02b6a702-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.091232 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.091190 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.136687 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.135049 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:36.136687 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.135078 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-76d99668b4-2x4pv"] Jul 07 20:02:36.155755 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.155721 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-nd79v"] Jul 07 20:02:36.172331 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:02:36.171802 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eede5f7_90c3_4d64_a6bc_ff8580fb3a33.slice/crio-76ace6fa221b1e77a425dbc48a48ccc43739093a95e231af9683011a712bd3bd WatchSource:0}: Error finding container 76ace6fa221b1e77a425dbc48a48ccc43739093a95e231af9683011a712bd3bd: Status 404 returned error can't find the container with id 76ace6fa221b1e77a425dbc48a48ccc43739093a95e231af9683011a712bd3bd Jul 07 20:02:36.172772 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:02:36.172556 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7301ae59_b24d_4bda_a5c0_7b8036966f8e.slice/crio-7a802762fee16ae636ee30707a73bbbbe03faa0b99d8e22d8e2e21f4dc7d230b WatchSource:0}: Error finding container 7a802762fee16ae636ee30707a73bbbbe03faa0b99d8e22d8e2e21f4dc7d230b: Status 404 returned error can't find the container with id 7a802762fee16ae636ee30707a73bbbbe03faa0b99d8e22d8e2e21f4dc7d230b Jul 07 20:02:36.429008 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:36.428904 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 20:02:36.434987 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:02:36.434878 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb45973a_992c_47af_ba0b_201f02b6a702.slice/crio-bce22afd994b51f04139041d4ce1dff18a58d50164aebc6d3c8304e81fedec64 WatchSource:0}: Error finding container bce22afd994b51f04139041d4ce1dff18a58d50164aebc6d3c8304e81fedec64: Status 404 returned error can't find the container with id bce22afd994b51f04139041d4ce1dff18a58d50164aebc6d3c8304e81fedec64 Jul 07 20:02:37.017013 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:37.016973 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zr7c2" event={"ID":"17142129-e574-4092-bd8d-0ffa3de6507a","Type":"ContainerStarted","Data":"188fe497f18b4c94ef8e7d209052a0cbb1466083f745119c8a3e3aeee2b1631b"} Jul 07 20:02:37.019611 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:37.019529 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-bsdc6" event={"ID":"1216713e-ac05-4a9b-94d0-5ec9e0caad6a","Type":"ContainerStarted","Data":"d91845d3d11fe7c982cf0da453f59b1a112f80ece9608bd5afc18aa7d5b4d854"} Jul 07 20:02:37.019611 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:37.019588 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-bsdc6" event={"ID":"1216713e-ac05-4a9b-94d0-5ec9e0caad6a","Type":"ContainerStarted","Data":"8127470fd1ad1f367aede862ea1ccc0531f5b77e86320d9ab536981f8e22e384"} Jul 07 20:02:37.019802 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:37.019617 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-bsdc6" event={"ID":"1216713e-ac05-4a9b-94d0-5ec9e0caad6a","Type":"ContainerStarted","Data":"705eaac3738c124ddb19363af7b4239d59bd7873bfbf8c2ebdea30fda56229dc"} Jul 07 20:02:37.022596 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:37.022280 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-nd79v" event={"ID":"7301ae59-b24d-4bda-a5c0-7b8036966f8e","Type":"ContainerStarted","Data":"7a802762fee16ae636ee30707a73bbbbe03faa0b99d8e22d8e2e21f4dc7d230b"} Jul 07 20:02:37.025038 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:37.025011 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-bsssh" event={"ID":"cab569a8-8055-4e0f-bba9-c698e829a799","Type":"ContainerStarted","Data":"9693127fead26ae0919903dc07dda69cb2b3909aac05551f7defc0b189f264d4"} Jul 07 20:02:37.027374 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:37.027337 2565 generic.go:356] "Generic (PLEG): container finished" podID="bb45973a-992c-47af-ba0b-201f02b6a702" containerID="7eaa6ccf73a0e21ce33a495387d3d21b18deaa1310059a83ec779d36336cf3a1" exitCode=0 Jul 07 20:02:37.027500 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:37.027431 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb45973a-992c-47af-ba0b-201f02b6a702","Type":"ContainerDied","Data":"7eaa6ccf73a0e21ce33a495387d3d21b18deaa1310059a83ec779d36336cf3a1"} Jul 07 20:02:37.027500 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:37.027492 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb45973a-992c-47af-ba0b-201f02b6a702","Type":"ContainerStarted","Data":"bce22afd994b51f04139041d4ce1dff18a58d50164aebc6d3c8304e81fedec64"} Jul 07 20:02:37.030525 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:37.030477 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wj7w5" event={"ID":"92ea0d19-f83c-4b0d-969d-cf288f52c55a","Type":"ContainerStarted","Data":"e530b361e69ceff42e72cac3cc89898dc9d2d2f2e6c6fd1565c5db81c9f11ab9"} Jul 07 20:02:37.030613 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:37.030543 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:02:37.032416 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:37.032379 2565 generic.go:356] "Generic (PLEG): container finished" podID="1f57d600-9387-4aa5-8798-1d7119506175" containerID="9930f977d974fe9e9c3ad29e7c33f12fa4d0c56a5b56e7b4e28cc0f5b88ee094" exitCode=0 Jul 07 20:02:37.032526 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:37.032472 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f57d600-9387-4aa5-8798-1d7119506175","Type":"ContainerDied","Data":"9930f977d974fe9e9c3ad29e7c33f12fa4d0c56a5b56e7b4e28cc0f5b88ee094"} Jul 07 20:02:37.035469 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:37.035422 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-76d99668b4-2x4pv" event={"ID":"6eede5f7-90c3-4d64-a6bc-ff8580fb3a33","Type":"ContainerStarted","Data":"76ace6fa221b1e77a425dbc48a48ccc43739093a95e231af9683011a712bd3bd"} Jul 07 20:02:37.039259 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:37.038689 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zr7c2" podStartSLOduration=66.847320807 podStartE2EDuration="1m15.038672774s" totalImagesPullingTime="8.191351967s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 20:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:02:37.038363781 +0000 UTC m=+77.192825733" watchObservedRunningTime="2026-07-07 20:02:37.038672774 +0000 UTC m=+77.193134704" Jul 07 20:02:37.067027 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:37.066930 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-bsdc6" podStartSLOduration=2.2529409129999998 podStartE2EDuration="8.066914881s" totalImagesPullingTime="5.813973968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:02:37.065735103 +0000 UTC m=+77.220197053" watchObservedRunningTime="2026-07-07 20:02:37.066914881 +0000 UTC m=+77.221376834" Jul 07 20:02:37.091729 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:37.091003 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-89snw" podStartSLOduration=6.945225955 podStartE2EDuration="8.090983436s" totalImagesPullingTime="1.065671763s" totalInitContainerRuntime="80.085718ms" isStatefulPod=true podCreationTimestamp="2026-07-07 20:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:02:37.090550584 +0000 UTC m=+77.245012537" watchObservedRunningTime="2026-07-07 20:02:37.090983436 +0000 UTC m=+77.245445394" Jul 07 20:02:37.112059 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:37.112012 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-bsssh" podStartSLOduration=3.783476759 podStartE2EDuration="8.111993897s" totalImagesPullingTime="4.328517138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:02:37.111393894 +0000 UTC m=+77.265855850" watchObservedRunningTime="2026-07-07 20:02:37.111993897 +0000 UTC m=+77.266455848" Jul 07 20:02:37.200872 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:37.200824 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wj7w5" podStartSLOduration=68.022922484 podStartE2EDuration="1m16.200808466s" totalImagesPullingTime="8.177885982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:01:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:02:37.199891362 +0000 UTC m=+77.354353312" watchObservedRunningTime="2026-07-07 20:02:37.200808466 +0000 UTC m=+77.355270414" Jul 07 20:02:39.047566 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:39.047520 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" event={"ID":"1a5cb3fd-134b-4b74-b687-8b05bdeb31b6","Type":"ContainerStarted","Data":"96b9d8276069ba2e9008704451886930d50b56865b45484fa5d7a45c62f72b84"} Jul 07 20:02:39.048044 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:39.047620 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" event={"ID":"1a5cb3fd-134b-4b74-b687-8b05bdeb31b6","Type":"ContainerStarted","Data":"5369f8800c6ffcacb7f169f03ad684a2660bd19d0f64ab738334f5cddfda0779"} Jul 07 20:02:39.048044 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:39.047666 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" event={"ID":"1a5cb3fd-134b-4b74-b687-8b05bdeb31b6","Type":"ContainerStarted","Data":"2246efdf69c758dbc423964e8556fe682b7ff749da83c27d3275ad337d16ae76"} Jul 07 20:02:39.049279 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:39.049251 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-nd79v" event={"ID":"7301ae59-b24d-4bda-a5c0-7b8036966f8e","Type":"ContainerStarted","Data":"fc8099a6c1836e9183114cfc02344c3e7b21d4ba75286e67275b21bf7a269e58"} Jul 07 20:02:39.049527 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:39.049472 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-nd79v" Jul 07 20:02:39.051300 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:39.051229 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-76d99668b4-2x4pv" event={"ID":"6eede5f7-90c3-4d64-a6bc-ff8580fb3a33","Type":"ContainerStarted","Data":"083651f2484bf488aa0c6d3ff019f8d53eafd99b471c87a7850c7c6f05868f21"} Jul 07 20:02:39.056012 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:39.055985 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-nd79v" Jul 07 20:02:39.081426 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:39.080585 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-nd79v" podStartSLOduration=2.60388029 podStartE2EDuration="5.080565806s" totalImagesPullingTime="2.476685516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:02:39.078878991 +0000 UTC m=+79.233340940" watchObservedRunningTime="2026-07-07 20:02:39.080565806 +0000 UTC m=+79.235027752" Jul 07 20:02:39.138673 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:39.138573 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-76d99668b4-2x4pv" podStartSLOduration=3.662713444 podStartE2EDuration="6.138557035s" totalImagesPullingTime="2.475843591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:02:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:02:39.136059358 +0000 UTC m=+79.290521308" watchObservedRunningTime="2026-07-07 20:02:39.138557035 +0000 UTC m=+79.293018982" Jul 07 20:02:40.058275 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:40.058239 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" event={"ID":"1a5cb3fd-134b-4b74-b687-8b05bdeb31b6","Type":"ContainerStarted","Data":"0d7fa3befefd25223411c1d9410175c4d166398a6fb4373ce6078847592a5f95"} Jul 07 20:02:40.058586 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:40.058322 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" event={"ID":"1a5cb3fd-134b-4b74-b687-8b05bdeb31b6","Type":"ContainerStarted","Data":"1ce6bd0bfcd3c33d8eb603cefc6d1690abb150b4c6d0379c15594286f2fc9106"} Jul 07 20:02:40.058586 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:40.058363 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" event={"ID":"1a5cb3fd-134b-4b74-b687-8b05bdeb31b6","Type":"ContainerStarted","Data":"85931854cf72b72036128558241d6c337cf259d2bcdd181254fbf6dd967636b1"} Jul 07 20:02:40.087356 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:40.087308 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" podStartSLOduration=2.955363476 podStartE2EDuration="9.087288365s" totalImagesPullingTime="6.131924889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:02:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:02:40.085601451 +0000 UTC m=+80.240063401" watchObservedRunningTime="2026-07-07 20:02:40.087288365 +0000 UTC m=+80.241750313" Jul 07 20:02:41.065198 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:41.065158 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f57d600-9387-4aa5-8798-1d7119506175","Type":"ContainerStarted","Data":"42813a5ea815aab20f5c14d613b64cffbcecd079390a5679ad7e170a61f0c492"} Jul 07 20:02:41.065547 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:41.065270 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f57d600-9387-4aa5-8798-1d7119506175","Type":"ContainerStarted","Data":"887121088aca2c4abe1727e41efde2299d47ccf98171739bf212e6cab889e4b0"} Jul 07 20:02:41.065547 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:41.065304 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f57d600-9387-4aa5-8798-1d7119506175","Type":"ContainerStarted","Data":"d1850f44fd0abfde4b5addc578441c4122b83bf2ea81b7a151782e213cf0a4f6"} Jul 07 20:02:41.065547 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:41.065342 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f57d600-9387-4aa5-8798-1d7119506175","Type":"ContainerStarted","Data":"d91f61e4e4aa76da37db97176ea90771cc6ec130c24aae04301f94e7cea6b39f"} Jul 07 20:02:41.065666 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:41.065643 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" Jul 07 20:02:41.065718 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:41.065699 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f57d600-9387-4aa5-8798-1d7119506175","Type":"ContainerStarted","Data":"b410a1c23d0d1a46a2d989f4cda7cae2573494b89db08ae85db1755e86d06453"} Jul 07 20:02:41.065758 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:41.065745 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f57d600-9387-4aa5-8798-1d7119506175","Type":"ContainerStarted","Data":"e7de534b49a8e006739402e1922d8e07350f7946d1d2c19ea2dc5f7d0908c874"} Jul 07 20:02:41.101118 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:41.101035 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.818197131 podStartE2EDuration="11.101018499s" totalImagesPullingTime="8.258023383s" totalInitContainerRuntime="24.797985ms" isStatefulPod=false podCreationTimestamp="2026-07-07 20:02:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:02:41.099321742 +0000 UTC m=+81.253783693" watchObservedRunningTime="2026-07-07 20:02:41.101018499 +0000 UTC m=+81.255480448" Jul 07 20:02:43.078283 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:43.078246 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb45973a-992c-47af-ba0b-201f02b6a702","Type":"ContainerStarted","Data":"27e03d112684563ccc1e53755f2071ff86d8d9384c24556d54c16c3d6d9c3540"} Jul 07 20:02:43.078730 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:43.078330 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb45973a-992c-47af-ba0b-201f02b6a702","Type":"ContainerStarted","Data":"41c5759faa072ca19fd8f607fc6689d97213f41f5b58e28d98f33bcb57b72437"} Jul 07 20:02:43.078730 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:43.078377 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb45973a-992c-47af-ba0b-201f02b6a702","Type":"ContainerStarted","Data":"362393fafe0dfbe3caae79bba2586a60620f60271d4c4ccb3e1150bc8210f7da"} Jul 07 20:02:43.078730 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:43.078423 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb45973a-992c-47af-ba0b-201f02b6a702","Type":"ContainerStarted","Data":"4e167949e71deca9d60b30937d04931c8b92c2a5a8615f803ebf55c802afca7c"} Jul 07 20:02:43.078730 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:43.078454 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb45973a-992c-47af-ba0b-201f02b6a702","Type":"ContainerStarted","Data":"4138874affe53a5582211cf1808da2b86474b0081c9497531098fa96d129f5ca"} Jul 07 20:02:43.078730 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:43.078482 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb45973a-992c-47af-ba0b-201f02b6a702","Type":"ContainerStarted","Data":"193c013305c398f4cea2345e0f6a60cfec4a705db2f6a09501b6285f5bae2382"} Jul 07 20:02:43.117223 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:43.117153 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.88360644 podStartE2EDuration="8.117120338s" totalImagesPullingTime="5.212071622s" totalInitContainerRuntime="21.442276ms" isStatefulPod=false podCreationTimestamp="2026-07-07 20:02:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:02:43.114577195 +0000 UTC m=+83.269039144" watchObservedRunningTime="2026-07-07 20:02:43.117120338 +0000 UTC m=+83.271582286" Jul 07 20:02:46.135975 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:46.135935 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:02:47.076966 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:47.076933 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-677c97cfd9-8jl49" Jul 07 20:02:54.142936 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:54.142884 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-76d99668b4-2x4pv" Jul 07 20:02:54.143442 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:02:54.142965 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-76d99668b4-2x4pv" Jul 07 20:03:08.042885 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:08.042856 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wj7w5" Jul 07 20:03:14.148048 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:14.148018 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-76d99668b4-2x4pv" Jul 07 20:03:14.151832 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:14.151809 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-76d99668b4-2x4pv" Jul 07 20:03:36.135732 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:36.135695 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:36.155307 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:36.155283 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:36.298367 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:36.298341 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:49.873526 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:49.873484 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 20:03:49.873990 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:49.873857 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1f57d600-9387-4aa5-8798-1d7119506175" containerName="alertmanager" containerID="cri-o://e7de534b49a8e006739402e1922d8e07350f7946d1d2c19ea2dc5f7d0908c874" gracePeriod=120 Jul 07 20:03:49.873990 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:49.873917 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1f57d600-9387-4aa5-8798-1d7119506175" containerName="kube-rbac-proxy-metric" containerID="cri-o://887121088aca2c4abe1727e41efde2299d47ccf98171739bf212e6cab889e4b0" gracePeriod=120 Jul 07 20:03:49.874107 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:49.873942 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1f57d600-9387-4aa5-8798-1d7119506175" containerName="kube-rbac-proxy-web" containerID="cri-o://d91f61e4e4aa76da37db97176ea90771cc6ec130c24aae04301f94e7cea6b39f" gracePeriod=120 Jul 07 20:03:49.874107 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:49.873978 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1f57d600-9387-4aa5-8798-1d7119506175" containerName="kube-rbac-proxy" containerID="cri-o://d1850f44fd0abfde4b5addc578441c4122b83bf2ea81b7a151782e213cf0a4f6" gracePeriod=120 Jul 07 20:03:49.874107 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:49.873995 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1f57d600-9387-4aa5-8798-1d7119506175" containerName="config-reloader" containerID="cri-o://b410a1c23d0d1a46a2d989f4cda7cae2573494b89db08ae85db1755e86d06453" gracePeriod=120 Jul 07 20:03:49.874107 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:49.873958 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1f57d600-9387-4aa5-8798-1d7119506175" containerName="prom-label-proxy" containerID="cri-o://42813a5ea815aab20f5c14d613b64cffbcecd079390a5679ad7e170a61f0c492" gracePeriod=120 Jul 07 20:03:50.338983 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:50.338952 2565 generic.go:356] "Generic (PLEG): container finished" podID="1f57d600-9387-4aa5-8798-1d7119506175" containerID="42813a5ea815aab20f5c14d613b64cffbcecd079390a5679ad7e170a61f0c492" exitCode=0 Jul 07 20:03:50.338983 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:50.338975 2565 generic.go:356] "Generic (PLEG): container finished" podID="1f57d600-9387-4aa5-8798-1d7119506175" containerID="887121088aca2c4abe1727e41efde2299d47ccf98171739bf212e6cab889e4b0" exitCode=0 Jul 07 20:03:50.338983 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:50.338981 2565 generic.go:356] "Generic (PLEG): container finished" podID="1f57d600-9387-4aa5-8798-1d7119506175" containerID="d1850f44fd0abfde4b5addc578441c4122b83bf2ea81b7a151782e213cf0a4f6" exitCode=0 Jul 07 20:03:50.338983 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:50.338988 2565 generic.go:356] "Generic (PLEG): container finished" podID="1f57d600-9387-4aa5-8798-1d7119506175" containerID="b410a1c23d0d1a46a2d989f4cda7cae2573494b89db08ae85db1755e86d06453" exitCode=0 Jul 07 20:03:50.339231 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:50.338996 2565 generic.go:356] "Generic (PLEG): container finished" podID="1f57d600-9387-4aa5-8798-1d7119506175" containerID="e7de534b49a8e006739402e1922d8e07350f7946d1d2c19ea2dc5f7d0908c874" exitCode=0 Jul 07 20:03:50.339231 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:50.339021 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f57d600-9387-4aa5-8798-1d7119506175","Type":"ContainerDied","Data":"42813a5ea815aab20f5c14d613b64cffbcecd079390a5679ad7e170a61f0c492"} Jul 07 20:03:50.339231 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:50.339083 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f57d600-9387-4aa5-8798-1d7119506175","Type":"ContainerDied","Data":"887121088aca2c4abe1727e41efde2299d47ccf98171739bf212e6cab889e4b0"} Jul 07 20:03:50.339231 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:50.339111 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f57d600-9387-4aa5-8798-1d7119506175","Type":"ContainerDied","Data":"d1850f44fd0abfde4b5addc578441c4122b83bf2ea81b7a151782e213cf0a4f6"} Jul 07 20:03:50.339231 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:50.339136 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f57d600-9387-4aa5-8798-1d7119506175","Type":"ContainerDied","Data":"b410a1c23d0d1a46a2d989f4cda7cae2573494b89db08ae85db1755e86d06453"} Jul 07 20:03:50.339231 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:50.339159 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f57d600-9387-4aa5-8798-1d7119506175","Type":"ContainerDied","Data":"e7de534b49a8e006739402e1922d8e07350f7946d1d2c19ea2dc5f7d0908c874"} Jul 07 20:03:51.116744 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.116719 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.138960 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.138936 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f57d600-9387-4aa5-8798-1d7119506175-alertmanager-trusted-ca-bundle\") pod \"1f57d600-9387-4aa5-8798-1d7119506175\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " Jul 07 20:03:51.139076 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.138994 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-secret-alertmanager-kube-rbac-proxy-web\") pod \"1f57d600-9387-4aa5-8798-1d7119506175\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " Jul 07 20:03:51.139076 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.139024 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1f57d600-9387-4aa5-8798-1d7119506175-alertmanager-main-db\") pod \"1f57d600-9387-4aa5-8798-1d7119506175\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " Jul 07 20:03:51.139076 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.139072 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f57d600-9387-4aa5-8798-1d7119506175-metrics-client-ca\") pod \"1f57d600-9387-4aa5-8798-1d7119506175\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " Jul 07 20:03:51.139237 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.139101 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f57d600-9387-4aa5-8798-1d7119506175-tls-assets\") pod \"1f57d600-9387-4aa5-8798-1d7119506175\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " Jul 07 20:03:51.139237 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.139129 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-secret-alertmanager-kube-rbac-proxy\") pod \"1f57d600-9387-4aa5-8798-1d7119506175\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " Jul 07 20:03:51.139237 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.139171 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmxhc\" (UniqueName: \"kubernetes.io/projected/1f57d600-9387-4aa5-8798-1d7119506175-kube-api-access-tmxhc\") pod \"1f57d600-9387-4aa5-8798-1d7119506175\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " Jul 07 20:03:51.139237 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.139213 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-secret-alertmanager-kube-rbac-proxy-metric\") pod \"1f57d600-9387-4aa5-8798-1d7119506175\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " Jul 07 20:03:51.139451 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.139247 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f57d600-9387-4aa5-8798-1d7119506175-config-out\") pod \"1f57d600-9387-4aa5-8798-1d7119506175\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " Jul 07 20:03:51.139451 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.139274 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-config-volume\") pod \"1f57d600-9387-4aa5-8798-1d7119506175\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " Jul 07 20:03:51.139451 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.139300 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-cluster-tls-config\") pod \"1f57d600-9387-4aa5-8798-1d7119506175\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " Jul 07 20:03:51.139451 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.139313 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f57d600-9387-4aa5-8798-1d7119506175-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "1f57d600-9387-4aa5-8798-1d7119506175" (UID: "1f57d600-9387-4aa5-8798-1d7119506175"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 20:03:51.139451 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.139337 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-secret-alertmanager-main-tls\") pod \"1f57d600-9387-4aa5-8798-1d7119506175\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " Jul 07 20:03:51.139451 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.139394 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-web-config\") pod \"1f57d600-9387-4aa5-8798-1d7119506175\" (UID: \"1f57d600-9387-4aa5-8798-1d7119506175\") " Jul 07 20:03:51.139744 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.139681 2565 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f57d600-9387-4aa5-8798-1d7119506175-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:51.140236 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.139334 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f57d600-9387-4aa5-8798-1d7119506175-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "1f57d600-9387-4aa5-8798-1d7119506175" (UID: "1f57d600-9387-4aa5-8798-1d7119506175"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:03:51.140334 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.139931 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f57d600-9387-4aa5-8798-1d7119506175-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "1f57d600-9387-4aa5-8798-1d7119506175" (UID: "1f57d600-9387-4aa5-8798-1d7119506175"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 20:03:51.142430 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.141934 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "1f57d600-9387-4aa5-8798-1d7119506175" (UID: "1f57d600-9387-4aa5-8798-1d7119506175"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:03:51.142430 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.142295 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f57d600-9387-4aa5-8798-1d7119506175-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1f57d600-9387-4aa5-8798-1d7119506175" (UID: "1f57d600-9387-4aa5-8798-1d7119506175"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 20:03:51.142927 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.142547 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "1f57d600-9387-4aa5-8798-1d7119506175" (UID: "1f57d600-9387-4aa5-8798-1d7119506175"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:03:51.142927 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.142885 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f57d600-9387-4aa5-8798-1d7119506175-config-out" (OuterVolumeSpecName: "config-out") pod "1f57d600-9387-4aa5-8798-1d7119506175" (UID: "1f57d600-9387-4aa5-8798-1d7119506175"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:03:51.143357 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.143326 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f57d600-9387-4aa5-8798-1d7119506175-kube-api-access-tmxhc" (OuterVolumeSpecName: "kube-api-access-tmxhc") pod "1f57d600-9387-4aa5-8798-1d7119506175" (UID: "1f57d600-9387-4aa5-8798-1d7119506175"). InnerVolumeSpecName "kube-api-access-tmxhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 20:03:51.143897 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.143872 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "1f57d600-9387-4aa5-8798-1d7119506175" (UID: "1f57d600-9387-4aa5-8798-1d7119506175"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:03:51.144005 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.143980 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "1f57d600-9387-4aa5-8798-1d7119506175" (UID: "1f57d600-9387-4aa5-8798-1d7119506175"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:03:51.144099 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.144072 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-config-volume" (OuterVolumeSpecName: "config-volume") pod "1f57d600-9387-4aa5-8798-1d7119506175" (UID: "1f57d600-9387-4aa5-8798-1d7119506175"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:03:51.148106 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.148083 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "1f57d600-9387-4aa5-8798-1d7119506175" (UID: "1f57d600-9387-4aa5-8798-1d7119506175"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:03:51.156051 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.156025 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-web-config" (OuterVolumeSpecName: "web-config") pod "1f57d600-9387-4aa5-8798-1d7119506175" (UID: "1f57d600-9387-4aa5-8798-1d7119506175"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:03:51.240366 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.240288 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmxhc\" (UniqueName: \"kubernetes.io/projected/1f57d600-9387-4aa5-8798-1d7119506175-kube-api-access-tmxhc\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:51.240366 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.240324 2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:51.240366 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.240335 2565 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f57d600-9387-4aa5-8798-1d7119506175-config-out\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:51.240366 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.240343 2565 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-config-volume\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:51.240366 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.240351 2565 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-cluster-tls-config\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:51.240366 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.240360 2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-secret-alertmanager-main-tls\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:51.240366 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.240368 2565 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-web-config\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:51.240366 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.240378 2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:51.240713 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.240386 2565 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1f57d600-9387-4aa5-8798-1d7119506175-alertmanager-main-db\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:51.240713 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.240394 2565 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f57d600-9387-4aa5-8798-1d7119506175-metrics-client-ca\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:51.240713 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.240426 2565 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f57d600-9387-4aa5-8798-1d7119506175-tls-assets\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:51.240713 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.240434 2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f57d600-9387-4aa5-8798-1d7119506175-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:51.346162 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.346125 2565 generic.go:356] "Generic (PLEG): container finished" podID="1f57d600-9387-4aa5-8798-1d7119506175" containerID="d91f61e4e4aa76da37db97176ea90771cc6ec130c24aae04301f94e7cea6b39f" exitCode=0 Jul 07 20:03:51.346309 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.346173 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f57d600-9387-4aa5-8798-1d7119506175","Type":"ContainerDied","Data":"d91f61e4e4aa76da37db97176ea90771cc6ec130c24aae04301f94e7cea6b39f"} Jul 07 20:03:51.346309 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.346213 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.346309 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.346247 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f57d600-9387-4aa5-8798-1d7119506175","Type":"ContainerDied","Data":"49f31fe07412ced946aa6c14d8faafcd584d529c94a8bf92e7f80fffad3c4727"} Jul 07 20:03:51.346309 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.346296 2565 scope.go:117] "RemoveContainer" containerID="42813a5ea815aab20f5c14d613b64cffbcecd079390a5679ad7e170a61f0c492" Jul 07 20:03:51.356282 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.356257 2565 scope.go:117] "RemoveContainer" containerID="887121088aca2c4abe1727e41efde2299d47ccf98171739bf212e6cab889e4b0" Jul 07 20:03:51.362934 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.362908 2565 scope.go:117] "RemoveContainer" containerID="d1850f44fd0abfde4b5addc578441c4122b83bf2ea81b7a151782e213cf0a4f6" Jul 07 20:03:51.369528 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.369507 2565 scope.go:117] "RemoveContainer" containerID="d91f61e4e4aa76da37db97176ea90771cc6ec130c24aae04301f94e7cea6b39f" Jul 07 20:03:51.376355 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.376320 2565 scope.go:117] "RemoveContainer" containerID="b410a1c23d0d1a46a2d989f4cda7cae2573494b89db08ae85db1755e86d06453" Jul 07 20:03:51.377452 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.377388 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 20:03:51.382444 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.382424 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 20:03:51.383875 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.383863 2565 scope.go:117] "RemoveContainer" containerID="e7de534b49a8e006739402e1922d8e07350f7946d1d2c19ea2dc5f7d0908c874" Jul 07 20:03:51.390367 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.390349 2565 scope.go:117] "RemoveContainer" containerID="9930f977d974fe9e9c3ad29e7c33f12fa4d0c56a5b56e7b4e28cc0f5b88ee094" Jul 07 20:03:51.396822 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.396805 2565 scope.go:117] "RemoveContainer" containerID="42813a5ea815aab20f5c14d613b64cffbcecd079390a5679ad7e170a61f0c492" Jul 07 20:03:51.397079 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:03:51.397046 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42813a5ea815aab20f5c14d613b64cffbcecd079390a5679ad7e170a61f0c492\": container with ID starting with 42813a5ea815aab20f5c14d613b64cffbcecd079390a5679ad7e170a61f0c492 not found: ID does not exist" containerID="42813a5ea815aab20f5c14d613b64cffbcecd079390a5679ad7e170a61f0c492" Jul 07 20:03:51.397121 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.397088 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42813a5ea815aab20f5c14d613b64cffbcecd079390a5679ad7e170a61f0c492"} err="failed to get container status \"42813a5ea815aab20f5c14d613b64cffbcecd079390a5679ad7e170a61f0c492\": rpc error: code = NotFound desc = could not find container \"42813a5ea815aab20f5c14d613b64cffbcecd079390a5679ad7e170a61f0c492\": container with ID starting with 42813a5ea815aab20f5c14d613b64cffbcecd079390a5679ad7e170a61f0c492 not found: ID does not exist" Jul 07 20:03:51.397165 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.397120 2565 scope.go:117] "RemoveContainer" containerID="887121088aca2c4abe1727e41efde2299d47ccf98171739bf212e6cab889e4b0" Jul 07 20:03:51.397350 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:03:51.397334 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"887121088aca2c4abe1727e41efde2299d47ccf98171739bf212e6cab889e4b0\": container with ID starting with 887121088aca2c4abe1727e41efde2299d47ccf98171739bf212e6cab889e4b0 not found: ID does not exist" containerID="887121088aca2c4abe1727e41efde2299d47ccf98171739bf212e6cab889e4b0" Jul 07 20:03:51.397423 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.397352 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"887121088aca2c4abe1727e41efde2299d47ccf98171739bf212e6cab889e4b0"} err="failed to get container status \"887121088aca2c4abe1727e41efde2299d47ccf98171739bf212e6cab889e4b0\": rpc error: code = NotFound desc = could not find container \"887121088aca2c4abe1727e41efde2299d47ccf98171739bf212e6cab889e4b0\": container with ID starting with 887121088aca2c4abe1727e41efde2299d47ccf98171739bf212e6cab889e4b0 not found: ID does not exist" Jul 07 20:03:51.397423 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.397363 2565 scope.go:117] "RemoveContainer" containerID="d1850f44fd0abfde4b5addc578441c4122b83bf2ea81b7a151782e213cf0a4f6" Jul 07 20:03:51.397614 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:03:51.397597 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1850f44fd0abfde4b5addc578441c4122b83bf2ea81b7a151782e213cf0a4f6\": container with ID starting with d1850f44fd0abfde4b5addc578441c4122b83bf2ea81b7a151782e213cf0a4f6 not found: ID does not exist" containerID="d1850f44fd0abfde4b5addc578441c4122b83bf2ea81b7a151782e213cf0a4f6" Jul 07 20:03:51.397651 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.397620 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1850f44fd0abfde4b5addc578441c4122b83bf2ea81b7a151782e213cf0a4f6"} err="failed to get container status \"d1850f44fd0abfde4b5addc578441c4122b83bf2ea81b7a151782e213cf0a4f6\": rpc error: code = NotFound desc = could not find container \"d1850f44fd0abfde4b5addc578441c4122b83bf2ea81b7a151782e213cf0a4f6\": container with ID starting with d1850f44fd0abfde4b5addc578441c4122b83bf2ea81b7a151782e213cf0a4f6 not found: ID does not exist" Jul 07 20:03:51.397651 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.397635 2565 scope.go:117] "RemoveContainer" containerID="d91f61e4e4aa76da37db97176ea90771cc6ec130c24aae04301f94e7cea6b39f" Jul 07 20:03:51.397863 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:03:51.397848 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d91f61e4e4aa76da37db97176ea90771cc6ec130c24aae04301f94e7cea6b39f\": container with ID starting with d91f61e4e4aa76da37db97176ea90771cc6ec130c24aae04301f94e7cea6b39f not found: ID does not exist" containerID="d91f61e4e4aa76da37db97176ea90771cc6ec130c24aae04301f94e7cea6b39f" Jul 07 20:03:51.397911 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.397868 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d91f61e4e4aa76da37db97176ea90771cc6ec130c24aae04301f94e7cea6b39f"} err="failed to get container status \"d91f61e4e4aa76da37db97176ea90771cc6ec130c24aae04301f94e7cea6b39f\": rpc error: code = NotFound desc = could not find container \"d91f61e4e4aa76da37db97176ea90771cc6ec130c24aae04301f94e7cea6b39f\": container with ID starting with d91f61e4e4aa76da37db97176ea90771cc6ec130c24aae04301f94e7cea6b39f not found: ID does not exist" Jul 07 20:03:51.397911 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.397882 2565 scope.go:117] "RemoveContainer" containerID="b410a1c23d0d1a46a2d989f4cda7cae2573494b89db08ae85db1755e86d06453" Jul 07 20:03:51.398090 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:03:51.398073 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b410a1c23d0d1a46a2d989f4cda7cae2573494b89db08ae85db1755e86d06453\": container with ID starting with b410a1c23d0d1a46a2d989f4cda7cae2573494b89db08ae85db1755e86d06453 not found: ID does not exist" containerID="b410a1c23d0d1a46a2d989f4cda7cae2573494b89db08ae85db1755e86d06453" Jul 07 20:03:51.398126 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.398096 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b410a1c23d0d1a46a2d989f4cda7cae2573494b89db08ae85db1755e86d06453"} err="failed to get container status \"b410a1c23d0d1a46a2d989f4cda7cae2573494b89db08ae85db1755e86d06453\": rpc error: code = NotFound desc = could not find container \"b410a1c23d0d1a46a2d989f4cda7cae2573494b89db08ae85db1755e86d06453\": container with ID starting with b410a1c23d0d1a46a2d989f4cda7cae2573494b89db08ae85db1755e86d06453 not found: ID does not exist" Jul 07 20:03:51.398126 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.398115 2565 scope.go:117] "RemoveContainer" containerID="e7de534b49a8e006739402e1922d8e07350f7946d1d2c19ea2dc5f7d0908c874" Jul 07 20:03:51.398340 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:03:51.398325 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7de534b49a8e006739402e1922d8e07350f7946d1d2c19ea2dc5f7d0908c874\": container with ID starting with e7de534b49a8e006739402e1922d8e07350f7946d1d2c19ea2dc5f7d0908c874 not found: ID does not exist" containerID="e7de534b49a8e006739402e1922d8e07350f7946d1d2c19ea2dc5f7d0908c874" Jul 07 20:03:51.398393 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.398343 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7de534b49a8e006739402e1922d8e07350f7946d1d2c19ea2dc5f7d0908c874"} err="failed to get container status \"e7de534b49a8e006739402e1922d8e07350f7946d1d2c19ea2dc5f7d0908c874\": rpc error: code = NotFound desc = could not find container \"e7de534b49a8e006739402e1922d8e07350f7946d1d2c19ea2dc5f7d0908c874\": container with ID starting with e7de534b49a8e006739402e1922d8e07350f7946d1d2c19ea2dc5f7d0908c874 not found: ID does not exist" Jul 07 20:03:51.398393 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.398355 2565 scope.go:117] "RemoveContainer" containerID="9930f977d974fe9e9c3ad29e7c33f12fa4d0c56a5b56e7b4e28cc0f5b88ee094" Jul 07 20:03:51.398555 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:03:51.398541 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9930f977d974fe9e9c3ad29e7c33f12fa4d0c56a5b56e7b4e28cc0f5b88ee094\": container with ID starting with 9930f977d974fe9e9c3ad29e7c33f12fa4d0c56a5b56e7b4e28cc0f5b88ee094 not found: ID does not exist" containerID="9930f977d974fe9e9c3ad29e7c33f12fa4d0c56a5b56e7b4e28cc0f5b88ee094" Jul 07 20:03:51.398589 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.398557 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9930f977d974fe9e9c3ad29e7c33f12fa4d0c56a5b56e7b4e28cc0f5b88ee094"} err="failed to get container status \"9930f977d974fe9e9c3ad29e7c33f12fa4d0c56a5b56e7b4e28cc0f5b88ee094\": rpc error: code = NotFound desc = could not find container \"9930f977d974fe9e9c3ad29e7c33f12fa4d0c56a5b56e7b4e28cc0f5b88ee094\": container with ID starting with 9930f977d974fe9e9c3ad29e7c33f12fa4d0c56a5b56e7b4e28cc0f5b88ee094 not found: ID does not exist" Jul 07 20:03:51.418256 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.418233 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 20:03:51.418525 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.418513 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f57d600-9387-4aa5-8798-1d7119506175" containerName="prom-label-proxy" Jul 07 20:03:51.418576 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.418527 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f57d600-9387-4aa5-8798-1d7119506175" containerName="prom-label-proxy" Jul 07 20:03:51.418576 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.418537 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f57d600-9387-4aa5-8798-1d7119506175" containerName="alertmanager" Jul 07 20:03:51.418576 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.418542 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f57d600-9387-4aa5-8798-1d7119506175" containerName="alertmanager" Jul 07 20:03:51.418576 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.418552 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f57d600-9387-4aa5-8798-1d7119506175" containerName="kube-rbac-proxy" Jul 07 20:03:51.418576 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.418557 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f57d600-9387-4aa5-8798-1d7119506175" containerName="kube-rbac-proxy" Jul 07 20:03:51.418576 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.418563 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f57d600-9387-4aa5-8798-1d7119506175" containerName="init-config-reloader" Jul 07 20:03:51.418576 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.418568 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f57d600-9387-4aa5-8798-1d7119506175" containerName="init-config-reloader" Jul 07 20:03:51.418576 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.418576 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f57d600-9387-4aa5-8798-1d7119506175" containerName="kube-rbac-proxy-metric" Jul 07 20:03:51.418797 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.418580 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f57d600-9387-4aa5-8798-1d7119506175" containerName="kube-rbac-proxy-metric" Jul 07 20:03:51.418797 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.418593 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f57d600-9387-4aa5-8798-1d7119506175" containerName="kube-rbac-proxy-web" Jul 07 20:03:51.418797 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.418598 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f57d600-9387-4aa5-8798-1d7119506175" containerName="kube-rbac-proxy-web" Jul 07 20:03:51.418797 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.418605 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f57d600-9387-4aa5-8798-1d7119506175" containerName="config-reloader" Jul 07 20:03:51.418797 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.418610 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f57d600-9387-4aa5-8798-1d7119506175" containerName="config-reloader" Jul 07 20:03:51.418797 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.418650 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f57d600-9387-4aa5-8798-1d7119506175" containerName="config-reloader" Jul 07 20:03:51.418797 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.418657 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f57d600-9387-4aa5-8798-1d7119506175" containerName="kube-rbac-proxy-metric" Jul 07 20:03:51.418797 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.418667 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f57d600-9387-4aa5-8798-1d7119506175" containerName="prom-label-proxy" Jul 07 20:03:51.418797 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.418673 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f57d600-9387-4aa5-8798-1d7119506175" containerName="alertmanager" Jul 07 20:03:51.418797 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.418678 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f57d600-9387-4aa5-8798-1d7119506175" containerName="kube-rbac-proxy-web" Jul 07 20:03:51.418797 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.418685 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f57d600-9387-4aa5-8798-1d7119506175" containerName="kube-rbac-proxy" Jul 07 20:03:51.422112 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.422095 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.424762 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.424740 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 07 20:03:51.424855 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.424802 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 07 20:03:51.424915 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.424901 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hqvjf\"" Jul 07 20:03:51.424968 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.424946 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 07 20:03:51.425025 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.424991 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 07 20:03:51.425025 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.425012 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 07 20:03:51.425119 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.425061 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 07 20:03:51.425485 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.425466 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 07 20:03:51.430912 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.430894 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 07 20:03:51.433264 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.433250 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 07 20:03:51.437859 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.437840 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 20:03:51.442550 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.441545 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3f440cf2-76c9-4344-bffc-365865c94c5f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.442550 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.441585 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f440cf2-76c9-4344-bffc-365865c94c5f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.442550 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.441634 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3f440cf2-76c9-4344-bffc-365865c94c5f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.442550 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.441663 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3f440cf2-76c9-4344-bffc-365865c94c5f-web-config\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.442550 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.441689 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3f440cf2-76c9-4344-bffc-365865c94c5f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.442550 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.441709 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3f440cf2-76c9-4344-bffc-365865c94c5f-config-out\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.442550 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.441730 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spz2s\" (UniqueName: \"kubernetes.io/projected/3f440cf2-76c9-4344-bffc-365865c94c5f-kube-api-access-spz2s\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.442550 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.441777 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3f440cf2-76c9-4344-bffc-365865c94c5f-config-volume\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.442550 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.441800 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3f440cf2-76c9-4344-bffc-365865c94c5f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.442550 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.441839 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3f440cf2-76c9-4344-bffc-365865c94c5f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.442550 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.441860 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f440cf2-76c9-4344-bffc-365865c94c5f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.442550 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.441901 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3f440cf2-76c9-4344-bffc-365865c94c5f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.442550 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.441926 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3f440cf2-76c9-4344-bffc-365865c94c5f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.543125 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.543042 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3f440cf2-76c9-4344-bffc-365865c94c5f-config-volume\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.543125 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.543079 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3f440cf2-76c9-4344-bffc-365865c94c5f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.543296 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.543207 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3f440cf2-76c9-4344-bffc-365865c94c5f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.543296 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.543235 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f440cf2-76c9-4344-bffc-365865c94c5f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.543296 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.543269 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3f440cf2-76c9-4344-bffc-365865c94c5f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.543296 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.543287 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3f440cf2-76c9-4344-bffc-365865c94c5f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.543508 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.543331 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3f440cf2-76c9-4344-bffc-365865c94c5f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.543508 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.543351 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f440cf2-76c9-4344-bffc-365865c94c5f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.543508 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.543427 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3f440cf2-76c9-4344-bffc-365865c94c5f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.543508 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.543455 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3f440cf2-76c9-4344-bffc-365865c94c5f-web-config\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.543508 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.543484 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3f440cf2-76c9-4344-bffc-365865c94c5f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.543508 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.543505 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3f440cf2-76c9-4344-bffc-365865c94c5f-config-out\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.543809 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.543527 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spz2s\" (UniqueName: \"kubernetes.io/projected/3f440cf2-76c9-4344-bffc-365865c94c5f-kube-api-access-spz2s\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.543809 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.543722 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3f440cf2-76c9-4344-bffc-365865c94c5f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.544650 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.544625 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f440cf2-76c9-4344-bffc-365865c94c5f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.546094 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.546064 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f440cf2-76c9-4344-bffc-365865c94c5f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.546341 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.546263 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3f440cf2-76c9-4344-bffc-365865c94c5f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.546341 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.546269 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3f440cf2-76c9-4344-bffc-365865c94c5f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.546550 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.546459 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3f440cf2-76c9-4344-bffc-365865c94c5f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.546730 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.546699 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3f440cf2-76c9-4344-bffc-365865c94c5f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.546908 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.546888 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3f440cf2-76c9-4344-bffc-365865c94c5f-config-volume\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.546985 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.546964 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3f440cf2-76c9-4344-bffc-365865c94c5f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.547037 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.547021 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3f440cf2-76c9-4344-bffc-365865c94c5f-config-out\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.547082 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.547071 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3f440cf2-76c9-4344-bffc-365865c94c5f-web-config\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.547780 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.547764 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3f440cf2-76c9-4344-bffc-365865c94c5f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.553168 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.553151 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spz2s\" (UniqueName: \"kubernetes.io/projected/3f440cf2-76c9-4344-bffc-365865c94c5f-kube-api-access-spz2s\") pod \"alertmanager-main-0\" (UID: \"3f440cf2-76c9-4344-bffc-365865c94c5f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.687982 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.687954 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f57d600-9387-4aa5-8798-1d7119506175" path="/var/lib/kubelet/pods/1f57d600-9387-4aa5-8798-1d7119506175/volumes" Jul 07 20:03:51.731905 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.731888 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 20:03:51.876756 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:51.876726 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 20:03:51.879538 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:03:51.879509 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f440cf2_76c9_4344_bffc_365865c94c5f.slice/crio-fbf2556dac1c6604065482c26bba5633785063515ccae212cd62fa1277eb5164 WatchSource:0}: Error finding container fbf2556dac1c6604065482c26bba5633785063515ccae212cd62fa1277eb5164: Status 404 returned error can't find the container with id fbf2556dac1c6604065482c26bba5633785063515ccae212cd62fa1277eb5164 Jul 07 20:03:52.352547 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:52.352513 2565 generic.go:356] "Generic (PLEG): container finished" podID="3f440cf2-76c9-4344-bffc-365865c94c5f" containerID="53d50c0939a23fa7c29bc1e1085dfc6201c906f3d1987d333acc57c85ca47464" exitCode=0 Jul 07 20:03:52.352942 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:52.352557 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f440cf2-76c9-4344-bffc-365865c94c5f","Type":"ContainerDied","Data":"53d50c0939a23fa7c29bc1e1085dfc6201c906f3d1987d333acc57c85ca47464"} Jul 07 20:03:52.352942 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:52.352622 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f440cf2-76c9-4344-bffc-365865c94c5f","Type":"ContainerStarted","Data":"fbf2556dac1c6604065482c26bba5633785063515ccae212cd62fa1277eb5164"} Jul 07 20:03:53.360418 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:53.360371 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f440cf2-76c9-4344-bffc-365865c94c5f","Type":"ContainerStarted","Data":"546e8f2ea6954f03fdd4169c941020909c0c13eace172890be9e239425b20212"} Jul 07 20:03:53.360767 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:53.360452 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f440cf2-76c9-4344-bffc-365865c94c5f","Type":"ContainerStarted","Data":"03a7b77f43fc09e170f5a97143d78f19952462f89ce276805efbc82792bf9411"} Jul 07 20:03:53.360767 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:53.360478 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f440cf2-76c9-4344-bffc-365865c94c5f","Type":"ContainerStarted","Data":"2a623d990676ac1781e30a172d8533011a460d1db067663cfad0f150a88b72cf"} Jul 07 20:03:53.360767 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:53.360503 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f440cf2-76c9-4344-bffc-365865c94c5f","Type":"ContainerStarted","Data":"cdb1d9cc366b5d30522dfb496d6d8c2d8d904947ea5d4a63f35538e99272180a"} Jul 07 20:03:53.360767 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:53.360526 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f440cf2-76c9-4344-bffc-365865c94c5f","Type":"ContainerStarted","Data":"3a34e2475ad07c70d7f63d12f9cbe61d99522e2f073a2a48134e366e35441106"} Jul 07 20:03:53.360767 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:53.360549 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f440cf2-76c9-4344-bffc-365865c94c5f","Type":"ContainerStarted","Data":"aedc6f0dcb2a87b1a8deb80b52919e2471e326c7fec7308e2c89349f85f9246b"} Jul 07 20:03:53.393607 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:53.393558 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.37799873 podStartE2EDuration="2.39353904s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.54031ms" isStatefulPod=false podCreationTimestamp="2026-07-07 20:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:03:53.390907326 +0000 UTC m=+153.545369290" watchObservedRunningTime="2026-07-07 20:03:53.39353904 +0000 UTC m=+153.548000990" Jul 07 20:03:53.897755 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:53.897716 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9"] Jul 07 20:03:53.900341 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:53.900320 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9" Jul 07 20:03:53.903008 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:53.902982 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 07 20:03:53.903125 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:53.902989 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 07 20:03:53.903183 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:53.903145 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 07 20:03:53.903387 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:53.903369 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-9x56f\"" Jul 07 20:03:53.903462 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:53.903442 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 07 20:03:53.903641 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:53.903626 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 07 20:03:53.909097 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:53.909079 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 07 20:03:53.915237 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:53.915216 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9"] Jul 07 20:03:53.960838 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:53.960807 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/333927dd-8b21-49c8-b36a-8050642aad71-secret-telemeter-client\") pod \"telemeter-client-58fdf88bf5-9kzj9\" (UID: \"333927dd-8b21-49c8-b36a-8050642aad71\") " pod="openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9" Jul 07 20:03:53.960838 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:53.960839 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/333927dd-8b21-49c8-b36a-8050642aad71-telemeter-client-tls\") pod \"telemeter-client-58fdf88bf5-9kzj9\" (UID: \"333927dd-8b21-49c8-b36a-8050642aad71\") " pod="openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9" Jul 07 20:03:53.961094 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:53.960871 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v47vd\" (UniqueName: \"kubernetes.io/projected/333927dd-8b21-49c8-b36a-8050642aad71-kube-api-access-v47vd\") pod \"telemeter-client-58fdf88bf5-9kzj9\" (UID: \"333927dd-8b21-49c8-b36a-8050642aad71\") " pod="openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9" Jul 07 20:03:53.961094 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:53.960921 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/333927dd-8b21-49c8-b36a-8050642aad71-telemeter-trusted-ca-bundle\") pod \"telemeter-client-58fdf88bf5-9kzj9\" (UID: \"333927dd-8b21-49c8-b36a-8050642aad71\") " pod="openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9" Jul 07 20:03:53.961094 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:53.960970 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/333927dd-8b21-49c8-b36a-8050642aad71-serving-certs-ca-bundle\") pod \"telemeter-client-58fdf88bf5-9kzj9\" (UID: \"333927dd-8b21-49c8-b36a-8050642aad71\") " pod="openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9" Jul 07 20:03:53.961094 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:53.961005 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/333927dd-8b21-49c8-b36a-8050642aad71-federate-client-tls\") pod \"telemeter-client-58fdf88bf5-9kzj9\" (UID: \"333927dd-8b21-49c8-b36a-8050642aad71\") " pod="openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9" Jul 07 20:03:53.961094 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:53.961076 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/333927dd-8b21-49c8-b36a-8050642aad71-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-58fdf88bf5-9kzj9\" (UID: \"333927dd-8b21-49c8-b36a-8050642aad71\") " pod="openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9" Jul 07 20:03:53.961312 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:53.961112 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/333927dd-8b21-49c8-b36a-8050642aad71-metrics-client-ca\") pod \"telemeter-client-58fdf88bf5-9kzj9\" (UID: \"333927dd-8b21-49c8-b36a-8050642aad71\") " pod="openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9" Jul 07 20:03:54.062318 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.062278 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/333927dd-8b21-49c8-b36a-8050642aad71-telemeter-trusted-ca-bundle\") pod \"telemeter-client-58fdf88bf5-9kzj9\" (UID: \"333927dd-8b21-49c8-b36a-8050642aad71\") " pod="openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9" Jul 07 20:03:54.062522 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.062339 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/333927dd-8b21-49c8-b36a-8050642aad71-serving-certs-ca-bundle\") pod \"telemeter-client-58fdf88bf5-9kzj9\" (UID: \"333927dd-8b21-49c8-b36a-8050642aad71\") " pod="openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9" Jul 07 20:03:54.062522 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.062363 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/333927dd-8b21-49c8-b36a-8050642aad71-federate-client-tls\") pod \"telemeter-client-58fdf88bf5-9kzj9\" (UID: \"333927dd-8b21-49c8-b36a-8050642aad71\") " pod="openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9" Jul 07 20:03:54.062522 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.062469 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/333927dd-8b21-49c8-b36a-8050642aad71-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-58fdf88bf5-9kzj9\" (UID: \"333927dd-8b21-49c8-b36a-8050642aad71\") " pod="openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9" Jul 07 20:03:54.062522 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.062519 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/333927dd-8b21-49c8-b36a-8050642aad71-metrics-client-ca\") pod \"telemeter-client-58fdf88bf5-9kzj9\" (UID: \"333927dd-8b21-49c8-b36a-8050642aad71\") " pod="openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9" Jul 07 20:03:54.062728 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.062544 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/333927dd-8b21-49c8-b36a-8050642aad71-secret-telemeter-client\") pod \"telemeter-client-58fdf88bf5-9kzj9\" (UID: \"333927dd-8b21-49c8-b36a-8050642aad71\") " pod="openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9" Jul 07 20:03:54.062728 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.062568 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/333927dd-8b21-49c8-b36a-8050642aad71-telemeter-client-tls\") pod \"telemeter-client-58fdf88bf5-9kzj9\" (UID: \"333927dd-8b21-49c8-b36a-8050642aad71\") " pod="openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9" Jul 07 20:03:54.062728 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.062614 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v47vd\" (UniqueName: \"kubernetes.io/projected/333927dd-8b21-49c8-b36a-8050642aad71-kube-api-access-v47vd\") pod \"telemeter-client-58fdf88bf5-9kzj9\" (UID: \"333927dd-8b21-49c8-b36a-8050642aad71\") " pod="openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9" Jul 07 20:03:54.063248 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.063211 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/333927dd-8b21-49c8-b36a-8050642aad71-telemeter-trusted-ca-bundle\") pod \"telemeter-client-58fdf88bf5-9kzj9\" (UID: \"333927dd-8b21-49c8-b36a-8050642aad71\") " pod="openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9" Jul 07 20:03:54.063737 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.063693 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/333927dd-8b21-49c8-b36a-8050642aad71-serving-certs-ca-bundle\") pod \"telemeter-client-58fdf88bf5-9kzj9\" (UID: \"333927dd-8b21-49c8-b36a-8050642aad71\") " pod="openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9" Jul 07 20:03:54.064117 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.064091 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/333927dd-8b21-49c8-b36a-8050642aad71-metrics-client-ca\") pod \"telemeter-client-58fdf88bf5-9kzj9\" (UID: \"333927dd-8b21-49c8-b36a-8050642aad71\") " pod="openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9" Jul 07 20:03:54.065784 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.065761 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/333927dd-8b21-49c8-b36a-8050642aad71-federate-client-tls\") pod \"telemeter-client-58fdf88bf5-9kzj9\" (UID: \"333927dd-8b21-49c8-b36a-8050642aad71\") " pod="openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9" Jul 07 20:03:54.065879 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.065791 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/333927dd-8b21-49c8-b36a-8050642aad71-secret-telemeter-client\") pod \"telemeter-client-58fdf88bf5-9kzj9\" (UID: \"333927dd-8b21-49c8-b36a-8050642aad71\") " pod="openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9" Jul 07 20:03:54.065936 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.065906 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/333927dd-8b21-49c8-b36a-8050642aad71-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-58fdf88bf5-9kzj9\" (UID: \"333927dd-8b21-49c8-b36a-8050642aad71\") " pod="openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9" Jul 07 20:03:54.065990 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.065953 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/333927dd-8b21-49c8-b36a-8050642aad71-telemeter-client-tls\") pod \"telemeter-client-58fdf88bf5-9kzj9\" (UID: \"333927dd-8b21-49c8-b36a-8050642aad71\") " pod="openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9" Jul 07 20:03:54.072608 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.072588 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v47vd\" (UniqueName: \"kubernetes.io/projected/333927dd-8b21-49c8-b36a-8050642aad71-kube-api-access-v47vd\") pod \"telemeter-client-58fdf88bf5-9kzj9\" (UID: \"333927dd-8b21-49c8-b36a-8050642aad71\") " pod="openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9" Jul 07 20:03:54.212557 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.212463 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 20:03:54.212727 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.212586 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9" Jul 07 20:03:54.212794 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.212750 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bb45973a-992c-47af-ba0b-201f02b6a702" containerName="prometheus" containerID="cri-o://193c013305c398f4cea2345e0f6a60cfec4a705db2f6a09501b6285f5bae2382" gracePeriod=600 Jul 07 20:03:54.213078 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.212806 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bb45973a-992c-47af-ba0b-201f02b6a702" containerName="kube-rbac-proxy" containerID="cri-o://41c5759faa072ca19fd8f607fc6689d97213f41f5b58e28d98f33bcb57b72437" gracePeriod=600 Jul 07 20:03:54.213078 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.212873 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bb45973a-992c-47af-ba0b-201f02b6a702" containerName="kube-rbac-proxy-web" containerID="cri-o://362393fafe0dfbe3caae79bba2586a60620f60271d4c4ccb3e1150bc8210f7da" gracePeriod=600 Jul 07 20:03:54.213078 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.212871 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bb45973a-992c-47af-ba0b-201f02b6a702" containerName="thanos-sidecar" containerID="cri-o://4e167949e71deca9d60b30937d04931c8b92c2a5a8615f803ebf55c802afca7c" gracePeriod=600 Jul 07 20:03:54.213078 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.212866 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bb45973a-992c-47af-ba0b-201f02b6a702" containerName="kube-rbac-proxy-thanos" containerID="cri-o://27e03d112684563ccc1e53755f2071ff86d8d9384c24556d54c16c3d6d9c3540" gracePeriod=600 Jul 07 20:03:54.213078 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.212862 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bb45973a-992c-47af-ba0b-201f02b6a702" containerName="config-reloader" containerID="cri-o://4138874affe53a5582211cf1808da2b86474b0081c9497531098fa96d129f5ca" gracePeriod=600 Jul 07 20:03:54.372703 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.372655 2565 generic.go:356] "Generic (PLEG): container finished" podID="bb45973a-992c-47af-ba0b-201f02b6a702" containerID="27e03d112684563ccc1e53755f2071ff86d8d9384c24556d54c16c3d6d9c3540" exitCode=0 Jul 07 20:03:54.372703 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.372684 2565 generic.go:356] "Generic (PLEG): container finished" podID="bb45973a-992c-47af-ba0b-201f02b6a702" containerID="41c5759faa072ca19fd8f607fc6689d97213f41f5b58e28d98f33bcb57b72437" exitCode=0 Jul 07 20:03:54.372703 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.372682 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb45973a-992c-47af-ba0b-201f02b6a702","Type":"ContainerDied","Data":"27e03d112684563ccc1e53755f2071ff86d8d9384c24556d54c16c3d6d9c3540"} Jul 07 20:03:54.372703 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.372693 2565 generic.go:356] "Generic (PLEG): container finished" podID="bb45973a-992c-47af-ba0b-201f02b6a702" containerID="362393fafe0dfbe3caae79bba2586a60620f60271d4c4ccb3e1150bc8210f7da" exitCode=0 Jul 07 20:03:54.372703 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.372702 2565 generic.go:356] "Generic (PLEG): container finished" podID="bb45973a-992c-47af-ba0b-201f02b6a702" containerID="4e167949e71deca9d60b30937d04931c8b92c2a5a8615f803ebf55c802afca7c" exitCode=0 Jul 07 20:03:54.372703 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.372709 2565 generic.go:356] "Generic (PLEG): container finished" podID="bb45973a-992c-47af-ba0b-201f02b6a702" containerID="4138874affe53a5582211cf1808da2b86474b0081c9497531098fa96d129f5ca" exitCode=0 Jul 07 20:03:54.372703 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.372717 2565 generic.go:356] "Generic (PLEG): container finished" podID="bb45973a-992c-47af-ba0b-201f02b6a702" containerID="193c013305c398f4cea2345e0f6a60cfec4a705db2f6a09501b6285f5bae2382" exitCode=0 Jul 07 20:03:54.373439 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.372766 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb45973a-992c-47af-ba0b-201f02b6a702","Type":"ContainerDied","Data":"41c5759faa072ca19fd8f607fc6689d97213f41f5b58e28d98f33bcb57b72437"} Jul 07 20:03:54.373439 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.372806 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb45973a-992c-47af-ba0b-201f02b6a702","Type":"ContainerDied","Data":"362393fafe0dfbe3caae79bba2586a60620f60271d4c4ccb3e1150bc8210f7da"} Jul 07 20:03:54.373439 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.372844 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb45973a-992c-47af-ba0b-201f02b6a702","Type":"ContainerDied","Data":"4e167949e71deca9d60b30937d04931c8b92c2a5a8615f803ebf55c802afca7c"} Jul 07 20:03:54.373439 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.372881 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb45973a-992c-47af-ba0b-201f02b6a702","Type":"ContainerDied","Data":"4138874affe53a5582211cf1808da2b86474b0081c9497531098fa96d129f5ca"} Jul 07 20:03:54.373439 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.372918 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb45973a-992c-47af-ba0b-201f02b6a702","Type":"ContainerDied","Data":"193c013305c398f4cea2345e0f6a60cfec4a705db2f6a09501b6285f5bae2382"} Jul 07 20:03:54.442880 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.442848 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9"] Jul 07 20:03:54.462504 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.462477 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:54.463748 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:03:54.463697 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod333927dd_8b21_49c8_b36a_8050642aad71.slice/crio-83268e981e794a38b183ccc4b879b7b903c398dc94af34ffc85f0d35d1780069 WatchSource:0}: Error finding container 83268e981e794a38b183ccc4b879b7b903c398dc94af34ffc85f0d35d1780069: Status 404 returned error can't find the container with id 83268e981e794a38b183ccc4b879b7b903c398dc94af34ffc85f0d35d1780069 Jul 07 20:03:54.565527 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.565501 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-configmap-kubelet-serving-ca-bundle\") pod \"bb45973a-992c-47af-ba0b-201f02b6a702\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " Jul 07 20:03:54.565527 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.565539 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-kube-rbac-proxy\") pod \"bb45973a-992c-47af-ba0b-201f02b6a702\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " Jul 07 20:03:54.565756 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.565565 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-prometheus-k8s-rulefiles-1\") pod \"bb45973a-992c-47af-ba0b-201f02b6a702\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " Jul 07 20:03:54.565756 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.565611 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-prometheus-k8s-rulefiles-0\") pod \"bb45973a-992c-47af-ba0b-201f02b6a702\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " Jul 07 20:03:54.565756 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.565636 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bb45973a-992c-47af-ba0b-201f02b6a702-tls-assets\") pod \"bb45973a-992c-47af-ba0b-201f02b6a702\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " Jul 07 20:03:54.565756 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.565662 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-configmap-metrics-client-ca\") pod \"bb45973a-992c-47af-ba0b-201f02b6a702\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " Jul 07 20:03:54.565756 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.565710 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bb45973a-992c-47af-ba0b-201f02b6a702-config-out\") pod \"bb45973a-992c-47af-ba0b-201f02b6a702\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " Jul 07 20:03:54.565756 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.565734 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhn7j\" (UniqueName: \"kubernetes.io/projected/bb45973a-992c-47af-ba0b-201f02b6a702-kube-api-access-xhn7j\") pod \"bb45973a-992c-47af-ba0b-201f02b6a702\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " Jul 07 20:03:54.566060 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.565760 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"bb45973a-992c-47af-ba0b-201f02b6a702\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " Jul 07 20:03:54.566060 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.565788 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-prometheus-k8s-rulefiles-2\") pod \"bb45973a-992c-47af-ba0b-201f02b6a702\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " Jul 07 20:03:54.566060 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.565821 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-configmap-serving-certs-ca-bundle\") pod \"bb45973a-992c-47af-ba0b-201f02b6a702\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " Jul 07 20:03:54.566060 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.565861 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-metrics-client-certs\") pod \"bb45973a-992c-47af-ba0b-201f02b6a702\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " Jul 07 20:03:54.566060 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.565887 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"bb45973a-992c-47af-ba0b-201f02b6a702\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " Jul 07 20:03:54.566060 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.565917 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-thanos-prometheus-http-client-file\") pod \"bb45973a-992c-47af-ba0b-201f02b6a702\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " Jul 07 20:03:54.566060 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.565948 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-prometheus-trusted-ca-bundle\") pod \"bb45973a-992c-47af-ba0b-201f02b6a702\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " Jul 07 20:03:54.566060 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.566000 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-config\") pod \"bb45973a-992c-47af-ba0b-201f02b6a702\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " Jul 07 20:03:54.566060 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.566034 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-grpc-tls\") pod \"bb45973a-992c-47af-ba0b-201f02b6a702\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " Jul 07 20:03:54.566575 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.566065 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-prometheus-k8s-tls\") pod \"bb45973a-992c-47af-ba0b-201f02b6a702\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " Jul 07 20:03:54.566575 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.566098 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bb45973a-992c-47af-ba0b-201f02b6a702-prometheus-k8s-db\") pod \"bb45973a-992c-47af-ba0b-201f02b6a702\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " Jul 07 20:03:54.566575 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.566124 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-web-config\") pod \"bb45973a-992c-47af-ba0b-201f02b6a702\" (UID: \"bb45973a-992c-47af-ba0b-201f02b6a702\") " Jul 07 20:03:54.568226 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.567412 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "bb45973a-992c-47af-ba0b-201f02b6a702" (UID: "bb45973a-992c-47af-ba0b-201f02b6a702"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 20:03:54.568226 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.567752 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "bb45973a-992c-47af-ba0b-201f02b6a702" (UID: "bb45973a-992c-47af-ba0b-201f02b6a702"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 20:03:54.568226 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.568161 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "bb45973a-992c-47af-ba0b-201f02b6a702" (UID: "bb45973a-992c-47af-ba0b-201f02b6a702"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 20:03:54.568807 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.568625 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "bb45973a-992c-47af-ba0b-201f02b6a702" (UID: "bb45973a-992c-47af-ba0b-201f02b6a702"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:03:54.569193 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.569167 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "bb45973a-992c-47af-ba0b-201f02b6a702" (UID: "bb45973a-992c-47af-ba0b-201f02b6a702"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 20:03:54.569482 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.569454 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "bb45973a-992c-47af-ba0b-201f02b6a702" (UID: "bb45973a-992c-47af-ba0b-201f02b6a702"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 20:03:54.570380 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.570353 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "bb45973a-992c-47af-ba0b-201f02b6a702" (UID: "bb45973a-992c-47af-ba0b-201f02b6a702"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 20:03:54.571595 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.571566 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb45973a-992c-47af-ba0b-201f02b6a702-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "bb45973a-992c-47af-ba0b-201f02b6a702" (UID: "bb45973a-992c-47af-ba0b-201f02b6a702"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:03:54.571744 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.571722 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "bb45973a-992c-47af-ba0b-201f02b6a702" (UID: "bb45973a-992c-47af-ba0b-201f02b6a702"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 20:03:54.572210 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.572182 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "bb45973a-992c-47af-ba0b-201f02b6a702" (UID: "bb45973a-992c-47af-ba0b-201f02b6a702"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:03:54.572725 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.572662 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "bb45973a-992c-47af-ba0b-201f02b6a702" (UID: "bb45973a-992c-47af-ba0b-201f02b6a702"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:03:54.573847 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.573810 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb45973a-992c-47af-ba0b-201f02b6a702-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "bb45973a-992c-47af-ba0b-201f02b6a702" (UID: "bb45973a-992c-47af-ba0b-201f02b6a702"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 20:03:54.574425 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.574386 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb45973a-992c-47af-ba0b-201f02b6a702-kube-api-access-xhn7j" (OuterVolumeSpecName: "kube-api-access-xhn7j") pod "bb45973a-992c-47af-ba0b-201f02b6a702" (UID: "bb45973a-992c-47af-ba0b-201f02b6a702"). InnerVolumeSpecName "kube-api-access-xhn7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 20:03:54.575089 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.575069 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb45973a-992c-47af-ba0b-201f02b6a702-config-out" (OuterVolumeSpecName: "config-out") pod "bb45973a-992c-47af-ba0b-201f02b6a702" (UID: "bb45973a-992c-47af-ba0b-201f02b6a702"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:03:54.575447 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.575387 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "bb45973a-992c-47af-ba0b-201f02b6a702" (UID: "bb45973a-992c-47af-ba0b-201f02b6a702"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:03:54.575694 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.575533 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-config" (OuterVolumeSpecName: "config") pod "bb45973a-992c-47af-ba0b-201f02b6a702" (UID: "bb45973a-992c-47af-ba0b-201f02b6a702"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:03:54.575923 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.575788 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "bb45973a-992c-47af-ba0b-201f02b6a702" (UID: "bb45973a-992c-47af-ba0b-201f02b6a702"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:03:54.575923 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.575900 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "bb45973a-992c-47af-ba0b-201f02b6a702" (UID: "bb45973a-992c-47af-ba0b-201f02b6a702"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:03:54.577191 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.577160 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "bb45973a-992c-47af-ba0b-201f02b6a702" (UID: "bb45973a-992c-47af-ba0b-201f02b6a702"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:03:54.585867 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.585839 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-web-config" (OuterVolumeSpecName: "web-config") pod "bb45973a-992c-47af-ba0b-201f02b6a702" (UID: "bb45973a-992c-47af-ba0b-201f02b6a702"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:03:54.667327 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.667291 2565 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-configmap-metrics-client-ca\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:54.667327 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.667320 2565 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bb45973a-992c-47af-ba0b-201f02b6a702-config-out\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:54.667327 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.667329 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhn7j\" (UniqueName: \"kubernetes.io/projected/bb45973a-992c-47af-ba0b-201f02b6a702-kube-api-access-xhn7j\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:54.667566 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.667339 2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:54.667566 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.667349 2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:54.667566 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.667357 2565 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:54.667566 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.667367 2565 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-metrics-client-certs\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:54.667566 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.667377 2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:54.667566 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.667386 2565 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:54.667566 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.667394 2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:54.667566 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.667420 2565 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-config\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:54.667566 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.667428 2565 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-grpc-tls\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:54.667566 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.667436 2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:54.667566 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.667443 2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bb45973a-992c-47af-ba0b-201f02b6a702-prometheus-k8s-db\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:54.667566 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.667453 2565 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-web-config\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:54.667566 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.667462 2565 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:54.667566 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.667470 2565 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bb45973a-992c-47af-ba0b-201f02b6a702-secret-kube-rbac-proxy\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:54.667566 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.667479 2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:54.667566 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.667488 2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bb45973a-992c-47af-ba0b-201f02b6a702-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:54.667566 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:54.667498 2565 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bb45973a-992c-47af-ba0b-201f02b6a702-tls-assets\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:03:55.379990 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.379907 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb45973a-992c-47af-ba0b-201f02b6a702","Type":"ContainerDied","Data":"bce22afd994b51f04139041d4ce1dff18a58d50164aebc6d3c8304e81fedec64"} Jul 07 20:03:55.380391 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.379917 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.380391 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.379993 2565 scope.go:117] "RemoveContainer" containerID="27e03d112684563ccc1e53755f2071ff86d8d9384c24556d54c16c3d6d9c3540" Jul 07 20:03:55.381271 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.380997 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9" event={"ID":"333927dd-8b21-49c8-b36a-8050642aad71","Type":"ContainerStarted","Data":"83268e981e794a38b183ccc4b879b7b903c398dc94af34ffc85f0d35d1780069"} Jul 07 20:03:55.387888 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.387871 2565 scope.go:117] "RemoveContainer" containerID="41c5759faa072ca19fd8f607fc6689d97213f41f5b58e28d98f33bcb57b72437" Jul 07 20:03:55.394616 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.394599 2565 scope.go:117] "RemoveContainer" containerID="362393fafe0dfbe3caae79bba2586a60620f60271d4c4ccb3e1150bc8210f7da" Jul 07 20:03:55.401444 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.401422 2565 scope.go:117] "RemoveContainer" containerID="4e167949e71deca9d60b30937d04931c8b92c2a5a8615f803ebf55c802afca7c" Jul 07 20:03:55.407955 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.407940 2565 scope.go:117] "RemoveContainer" containerID="4138874affe53a5582211cf1808da2b86474b0081c9497531098fa96d129f5ca" Jul 07 20:03:55.411629 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.410555 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 20:03:55.415766 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.415260 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 20:03:55.418598 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.418579 2565 scope.go:117] "RemoveContainer" containerID="193c013305c398f4cea2345e0f6a60cfec4a705db2f6a09501b6285f5bae2382" Jul 07 20:03:55.425210 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.425197 2565 scope.go:117] "RemoveContainer" containerID="7eaa6ccf73a0e21ce33a495387d3d21b18deaa1310059a83ec779d36336cf3a1" Jul 07 20:03:55.448245 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.448227 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 20:03:55.448647 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.448629 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb45973a-992c-47af-ba0b-201f02b6a702" containerName="init-config-reloader" Jul 07 20:03:55.448719 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.448651 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb45973a-992c-47af-ba0b-201f02b6a702" containerName="init-config-reloader" Jul 07 20:03:55.448719 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.448670 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb45973a-992c-47af-ba0b-201f02b6a702" containerName="thanos-sidecar" Jul 07 20:03:55.448719 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.448679 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb45973a-992c-47af-ba0b-201f02b6a702" containerName="thanos-sidecar" Jul 07 20:03:55.448719 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.448698 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb45973a-992c-47af-ba0b-201f02b6a702" containerName="kube-rbac-proxy-web" Jul 07 20:03:55.448719 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.448706 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb45973a-992c-47af-ba0b-201f02b6a702" containerName="kube-rbac-proxy-web" Jul 07 20:03:55.448719 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.448717 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb45973a-992c-47af-ba0b-201f02b6a702" containerName="kube-rbac-proxy-thanos" Jul 07 20:03:55.448938 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.448726 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb45973a-992c-47af-ba0b-201f02b6a702" containerName="kube-rbac-proxy-thanos" Jul 07 20:03:55.448938 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.448736 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb45973a-992c-47af-ba0b-201f02b6a702" containerName="config-reloader" Jul 07 20:03:55.448938 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.448744 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb45973a-992c-47af-ba0b-201f02b6a702" containerName="config-reloader" Jul 07 20:03:55.448938 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.448761 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb45973a-992c-47af-ba0b-201f02b6a702" containerName="kube-rbac-proxy" Jul 07 20:03:55.448938 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.448770 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb45973a-992c-47af-ba0b-201f02b6a702" containerName="kube-rbac-proxy" Jul 07 20:03:55.448938 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.448779 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb45973a-992c-47af-ba0b-201f02b6a702" containerName="prometheus" Jul 07 20:03:55.448938 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.448785 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb45973a-992c-47af-ba0b-201f02b6a702" containerName="prometheus" Jul 07 20:03:55.448938 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.448831 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb45973a-992c-47af-ba0b-201f02b6a702" containerName="thanos-sidecar" Jul 07 20:03:55.448938 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.448839 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb45973a-992c-47af-ba0b-201f02b6a702" containerName="prometheus" Jul 07 20:03:55.448938 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.448847 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb45973a-992c-47af-ba0b-201f02b6a702" containerName="kube-rbac-proxy-web" Jul 07 20:03:55.448938 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.448853 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb45973a-992c-47af-ba0b-201f02b6a702" containerName="config-reloader" Jul 07 20:03:55.448938 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.448858 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb45973a-992c-47af-ba0b-201f02b6a702" containerName="kube-rbac-proxy" Jul 07 20:03:55.448938 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.448863 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb45973a-992c-47af-ba0b-201f02b6a702" containerName="kube-rbac-proxy-thanos" Jul 07 20:03:55.452926 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.452892 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.459240 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.459214 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 07 20:03:55.459473 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.459452 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 07 20:03:55.459580 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.459452 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 07 20:03:55.459639 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.459587 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 07 20:03:55.462109 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.462089 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 07 20:03:55.462195 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.462105 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 07 20:03:55.462326 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.462311 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 07 20:03:55.462458 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.462441 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 07 20:03:55.462559 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.462541 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 07 20:03:55.464164 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.464146 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 07 20:03:55.464254 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.464182 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 07 20:03:55.464519 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.464498 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9f8o4jglddmpe\"" Jul 07 20:03:55.464591 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.464544 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fpfzr\"" Jul 07 20:03:55.464785 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.464764 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 07 20:03:55.466471 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.466455 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 07 20:03:55.472369 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.472350 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 07 20:03:55.472721 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.472701 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/535f2334-abec-4253-a96b-ea6f4d63a585-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.472816 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.472727 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/535f2334-abec-4253-a96b-ea6f4d63a585-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.472816 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.472749 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/535f2334-abec-4253-a96b-ea6f4d63a585-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.472816 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.472798 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/535f2334-abec-4253-a96b-ea6f4d63a585-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.472971 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.472841 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/535f2334-abec-4253-a96b-ea6f4d63a585-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.472971 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.472868 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/535f2334-abec-4253-a96b-ea6f4d63a585-config\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.472971 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.472885 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/535f2334-abec-4253-a96b-ea6f4d63a585-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.472971 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.472955 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/535f2334-abec-4253-a96b-ea6f4d63a585-web-config\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.473170 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.472994 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/535f2334-abec-4253-a96b-ea6f4d63a585-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.473170 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.473051 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/535f2334-abec-4253-a96b-ea6f4d63a585-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.473170 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.473068 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/535f2334-abec-4253-a96b-ea6f4d63a585-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.473170 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.473085 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/535f2334-abec-4253-a96b-ea6f4d63a585-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.473170 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.473139 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/535f2334-abec-4253-a96b-ea6f4d63a585-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.473416 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.473172 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/535f2334-abec-4253-a96b-ea6f4d63a585-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.473416 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.473196 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/535f2334-abec-4253-a96b-ea6f4d63a585-config-out\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.473416 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.473210 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/535f2334-abec-4253-a96b-ea6f4d63a585-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.473416 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.473227 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/535f2334-abec-4253-a96b-ea6f4d63a585-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.473416 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.473289 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/535f2334-abec-4253-a96b-ea6f4d63a585-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.473416 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.473330 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/535f2334-abec-4253-a96b-ea6f4d63a585-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.473416 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.473358 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6pbl\" (UniqueName: \"kubernetes.io/projected/535f2334-abec-4253-a96b-ea6f4d63a585-kube-api-access-v6pbl\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.482997 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.482973 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 20:03:55.574465 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.574416 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/535f2334-abec-4253-a96b-ea6f4d63a585-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.574638 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.574473 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/535f2334-abec-4253-a96b-ea6f4d63a585-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.574696 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.574661 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/535f2334-abec-4253-a96b-ea6f4d63a585-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.574812 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.574793 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6pbl\" (UniqueName: \"kubernetes.io/projected/535f2334-abec-4253-a96b-ea6f4d63a585-kube-api-access-v6pbl\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.574928 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.574869 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/535f2334-abec-4253-a96b-ea6f4d63a585-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.574928 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.574898 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/535f2334-abec-4253-a96b-ea6f4d63a585-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.575012 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.574936 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/535f2334-abec-4253-a96b-ea6f4d63a585-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.575012 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.574992 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/535f2334-abec-4253-a96b-ea6f4d63a585-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.575108 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.575021 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/535f2334-abec-4253-a96b-ea6f4d63a585-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.575108 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.575061 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/535f2334-abec-4253-a96b-ea6f4d63a585-config\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.575108 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.575092 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/535f2334-abec-4253-a96b-ea6f4d63a585-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.575256 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.575120 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/535f2334-abec-4253-a96b-ea6f4d63a585-web-config\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.575256 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.575146 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/535f2334-abec-4253-a96b-ea6f4d63a585-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.575256 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.575180 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/535f2334-abec-4253-a96b-ea6f4d63a585-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.575256 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.575211 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/535f2334-abec-4253-a96b-ea6f4d63a585-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.575256 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.575239 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/535f2334-abec-4253-a96b-ea6f4d63a585-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.575530 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.575270 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/535f2334-abec-4253-a96b-ea6f4d63a585-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.575530 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.575297 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/535f2334-abec-4253-a96b-ea6f4d63a585-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.575530 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.575326 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/535f2334-abec-4253-a96b-ea6f4d63a585-config-out\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.575530 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.575337 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/535f2334-abec-4253-a96b-ea6f4d63a585-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.575530 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.575354 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/535f2334-abec-4253-a96b-ea6f4d63a585-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.575530 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.575380 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/535f2334-abec-4253-a96b-ea6f4d63a585-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.575811 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.575586 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/535f2334-abec-4253-a96b-ea6f4d63a585-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.575811 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.575593 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/535f2334-abec-4253-a96b-ea6f4d63a585-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.578130 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.578100 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/535f2334-abec-4253-a96b-ea6f4d63a585-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.578252 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.578188 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/535f2334-abec-4253-a96b-ea6f4d63a585-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.578437 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.578391 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/535f2334-abec-4253-a96b-ea6f4d63a585-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.578770 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.578703 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/535f2334-abec-4253-a96b-ea6f4d63a585-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.578896 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.578785 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/535f2334-abec-4253-a96b-ea6f4d63a585-config\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.578896 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.578887 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/535f2334-abec-4253-a96b-ea6f4d63a585-config-out\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.579060 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.579016 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/535f2334-abec-4253-a96b-ea6f4d63a585-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.579712 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.579688 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/535f2334-abec-4253-a96b-ea6f4d63a585-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.579889 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.579866 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/535f2334-abec-4253-a96b-ea6f4d63a585-web-config\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.580980 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.580936 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/535f2334-abec-4253-a96b-ea6f4d63a585-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.580980 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.580958 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/535f2334-abec-4253-a96b-ea6f4d63a585-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.581481 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.581458 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/535f2334-abec-4253-a96b-ea6f4d63a585-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.581839 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.581816 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/535f2334-abec-4253-a96b-ea6f4d63a585-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.581931 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.581855 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/535f2334-abec-4253-a96b-ea6f4d63a585-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.582973 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.582955 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/535f2334-abec-4253-a96b-ea6f4d63a585-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.585338 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.585313 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6pbl\" (UniqueName: \"kubernetes.io/projected/535f2334-abec-4253-a96b-ea6f4d63a585-kube-api-access-v6pbl\") pod \"prometheus-k8s-0\" (UID: \"535f2334-abec-4253-a96b-ea6f4d63a585\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.684029 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.683987 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb45973a-992c-47af-ba0b-201f02b6a702" path="/var/lib/kubelet/pods/bb45973a-992c-47af-ba0b-201f02b6a702/volumes" Jul 07 20:03:55.764133 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.764098 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:03:55.924283 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:55.924253 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 20:03:55.927681 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:03:55.927655 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod535f2334_abec_4253_a96b_ea6f4d63a585.slice/crio-4ad4e6cf653803cb0678c983093514d93153dd5d9fbd11203b7217b66a43e7f3 WatchSource:0}: Error finding container 4ad4e6cf653803cb0678c983093514d93153dd5d9fbd11203b7217b66a43e7f3: Status 404 returned error can't find the container with id 4ad4e6cf653803cb0678c983093514d93153dd5d9fbd11203b7217b66a43e7f3 Jul 07 20:03:56.389870 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:56.389832 2565 generic.go:356] "Generic (PLEG): container finished" podID="535f2334-abec-4253-a96b-ea6f4d63a585" containerID="82da8f6ee8d673aed9d23e1720ee0cf89945ed64d4cd28f8f969e1bc507ff7e9" exitCode=0 Jul 07 20:03:56.390293 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:56.389912 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"535f2334-abec-4253-a96b-ea6f4d63a585","Type":"ContainerDied","Data":"82da8f6ee8d673aed9d23e1720ee0cf89945ed64d4cd28f8f969e1bc507ff7e9"} Jul 07 20:03:56.390293 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:56.389967 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"535f2334-abec-4253-a96b-ea6f4d63a585","Type":"ContainerStarted","Data":"4ad4e6cf653803cb0678c983093514d93153dd5d9fbd11203b7217b66a43e7f3"} Jul 07 20:03:57.398205 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:57.398167 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9" event={"ID":"333927dd-8b21-49c8-b36a-8050642aad71","Type":"ContainerStarted","Data":"30e0fc8a86b187ecab7edabfeb70b4b2e43e0e7221a0d7a2f2243f5638e5ae89"} Jul 07 20:03:57.398610 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:57.398255 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9" event={"ID":"333927dd-8b21-49c8-b36a-8050642aad71","Type":"ContainerStarted","Data":"e108ffdcaeca986b2ea3c03f89dbddd35592aed7ad1bbe2f32a7568b8fbb8e1a"} Jul 07 20:03:57.398610 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:57.398283 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9" event={"ID":"333927dd-8b21-49c8-b36a-8050642aad71","Type":"ContainerStarted","Data":"3a55d057427c4dd6304498d924dfd36559703bead387dbbf9295d9c13382c397"} Jul 07 20:03:57.401186 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:57.401157 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"535f2334-abec-4253-a96b-ea6f4d63a585","Type":"ContainerStarted","Data":"ae52d1b6e133e8f4fcab786c4724a9787346e06109851ba39c10339779aa482e"} Jul 07 20:03:57.401282 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:57.401214 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"535f2334-abec-4253-a96b-ea6f4d63a585","Type":"ContainerStarted","Data":"e05f841f41de8606db3d1052d3d89a7b204ac848ecd88a8f31403850b65a65a3"} Jul 07 20:03:57.401282 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:57.401246 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"535f2334-abec-4253-a96b-ea6f4d63a585","Type":"ContainerStarted","Data":"5f31d2360cce4ff4ae298b96e3cf2f86c3ed98163699dc87122ee9cc3275ec84"} Jul 07 20:03:57.401282 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:57.401274 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"535f2334-abec-4253-a96b-ea6f4d63a585","Type":"ContainerStarted","Data":"1e52d62f53c61a1e8fc6585fa62eacc361eb9c802dfab9783ccd6f9cf3a1573b"} Jul 07 20:03:57.401375 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:57.401303 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"535f2334-abec-4253-a96b-ea6f4d63a585","Type":"ContainerStarted","Data":"4f3254b61f1c61d7f994b499979305dad491c07c0bb7dbf283c7bb31aa5a4e8a"} Jul 07 20:03:57.401375 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:57.401333 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"535f2334-abec-4253-a96b-ea6f4d63a585","Type":"ContainerStarted","Data":"d5d49619e6ad3f2b8a3d49166cc14f4d92db1dd9bebfb4a436bb2baedd27380b"} Jul 07 20:03:57.426936 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:57.426885 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-58fdf88bf5-9kzj9" podStartSLOduration=2.277048586 podStartE2EDuration="4.426865291s" totalImagesPullingTime="2.149816705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:03:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:03:57.42664277 +0000 UTC m=+157.581104730" watchObservedRunningTime="2026-07-07 20:03:57.426865291 +0000 UTC m=+157.581327243" Jul 07 20:03:57.472557 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:03:57.472507 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.451386539 podStartE2EDuration="2.47249074s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.104201ms" isStatefulPod=false podCreationTimestamp="2026-07-07 20:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:03:57.469217075 +0000 UTC m=+157.623679024" watchObservedRunningTime="2026-07-07 20:03:57.47249074 +0000 UTC m=+157.626952724" Jul 07 20:04:00.764369 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:04:00.764330 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:04:55.764468 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:04:55.764431 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:04:55.779616 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:04:55.779593 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:04:56.651502 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:04:56.651471 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 20:06:21.508631 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:21.508603 2565 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 20:06:27.082148 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:27.082116 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2tvvd"] Jul 07 20:06:27.085293 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:27.085274 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2tvvd" Jul 07 20:06:27.087936 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:27.087916 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 07 20:06:27.088023 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:27.087946 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 07 20:06:27.088101 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:27.088084 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-jr2l9\"" Jul 07 20:06:27.095505 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:27.095485 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2tvvd"] Jul 07 20:06:27.122991 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:27.122962 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qbh9\" (UniqueName: \"kubernetes.io/projected/c46f97bd-49fd-411a-ac64-fb6a6a6e7fe3-kube-api-access-8qbh9\") pod \"cert-manager-cainjector-559cdd8c87-2tvvd\" (UID: \"c46f97bd-49fd-411a-ac64-fb6a6a6e7fe3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2tvvd" Jul 07 20:06:27.123114 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:27.123026 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c46f97bd-49fd-411a-ac64-fb6a6a6e7fe3-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2tvvd\" (UID: \"c46f97bd-49fd-411a-ac64-fb6a6a6e7fe3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2tvvd" Jul 07 20:06:27.223669 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:27.223628 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c46f97bd-49fd-411a-ac64-fb6a6a6e7fe3-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2tvvd\" (UID: \"c46f97bd-49fd-411a-ac64-fb6a6a6e7fe3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2tvvd" Jul 07 20:06:27.223827 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:27.223683 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qbh9\" (UniqueName: \"kubernetes.io/projected/c46f97bd-49fd-411a-ac64-fb6a6a6e7fe3-kube-api-access-8qbh9\") pod \"cert-manager-cainjector-559cdd8c87-2tvvd\" (UID: \"c46f97bd-49fd-411a-ac64-fb6a6a6e7fe3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2tvvd" Jul 07 20:06:27.232276 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:27.232242 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c46f97bd-49fd-411a-ac64-fb6a6a6e7fe3-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2tvvd\" (UID: \"c46f97bd-49fd-411a-ac64-fb6a6a6e7fe3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2tvvd" Jul 07 20:06:27.232685 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:27.232665 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qbh9\" (UniqueName: \"kubernetes.io/projected/c46f97bd-49fd-411a-ac64-fb6a6a6e7fe3-kube-api-access-8qbh9\") pod \"cert-manager-cainjector-559cdd8c87-2tvvd\" (UID: \"c46f97bd-49fd-411a-ac64-fb6a6a6e7fe3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2tvvd" Jul 07 20:06:27.409226 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:27.409198 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2tvvd" Jul 07 20:06:27.541852 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:27.541667 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2tvvd"] Jul 07 20:06:27.544773 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:06:27.544737 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc46f97bd_49fd_411a_ac64_fb6a6a6e7fe3.slice/crio-3de11a105ce727602816163bdf112b37ebde2faae2e6a36cd4efb248c1648f80 WatchSource:0}: Error finding container 3de11a105ce727602816163bdf112b37ebde2faae2e6a36cd4efb248c1648f80: Status 404 returned error can't find the container with id 3de11a105ce727602816163bdf112b37ebde2faae2e6a36cd4efb248c1648f80 Jul 07 20:06:27.547041 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:27.547024 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 20:06:27.983298 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:27.983267 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2tvvd" event={"ID":"c46f97bd-49fd-411a-ac64-fb6a6a6e7fe3","Type":"ContainerStarted","Data":"3de11a105ce727602816163bdf112b37ebde2faae2e6a36cd4efb248c1648f80"} Jul 07 20:06:30.997567 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:30.997530 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2tvvd" event={"ID":"c46f97bd-49fd-411a-ac64-fb6a6a6e7fe3","Type":"ContainerStarted","Data":"077e995d9f0f68ecb03f5913774c8eb98b4297716482cdb3921ff673e9e7b922"} Jul 07 20:06:31.015658 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:31.015610 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2tvvd" podStartSLOduration=1.09179941 podStartE2EDuration="4.015595761s" totalImagesPullingTime="2.923796351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:06:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:06:31.014028781 +0000 UTC m=+311.168490731" watchObservedRunningTime="2026-07-07 20:06:31.015595761 +0000 UTC m=+311.170057709" Jul 07 20:06:37.253101 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:37.253066 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xg82n"] Jul 07 20:06:37.256535 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:37.256519 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xg82n" Jul 07 20:06:37.259296 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:37.259260 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-zhxz9\"" Jul 07 20:06:37.259436 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:37.259299 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 07 20:06:37.259436 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:37.259336 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 07 20:06:37.265178 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:37.265156 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xg82n"] Jul 07 20:06:37.311057 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:37.311033 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f2f71c8-ba9b-43d4-8b14-6406ac50f2da-tmp\") pod \"openshift-lws-operator-fd8ccff4c-xg82n\" (UID: \"4f2f71c8-ba9b-43d4-8b14-6406ac50f2da\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xg82n" Jul 07 20:06:37.311170 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:37.311075 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsdsd\" (UniqueName: \"kubernetes.io/projected/4f2f71c8-ba9b-43d4-8b14-6406ac50f2da-kube-api-access-fsdsd\") pod \"openshift-lws-operator-fd8ccff4c-xg82n\" (UID: \"4f2f71c8-ba9b-43d4-8b14-6406ac50f2da\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xg82n" Jul 07 20:06:37.412262 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:37.412236 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f2f71c8-ba9b-43d4-8b14-6406ac50f2da-tmp\") pod \"openshift-lws-operator-fd8ccff4c-xg82n\" (UID: \"4f2f71c8-ba9b-43d4-8b14-6406ac50f2da\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xg82n" Jul 07 20:06:37.412424 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:37.412275 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsdsd\" (UniqueName: \"kubernetes.io/projected/4f2f71c8-ba9b-43d4-8b14-6406ac50f2da-kube-api-access-fsdsd\") pod \"openshift-lws-operator-fd8ccff4c-xg82n\" (UID: \"4f2f71c8-ba9b-43d4-8b14-6406ac50f2da\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xg82n" Jul 07 20:06:37.412636 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:37.412617 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f2f71c8-ba9b-43d4-8b14-6406ac50f2da-tmp\") pod \"openshift-lws-operator-fd8ccff4c-xg82n\" (UID: \"4f2f71c8-ba9b-43d4-8b14-6406ac50f2da\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xg82n" Jul 07 20:06:37.423715 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:37.423696 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsdsd\" (UniqueName: \"kubernetes.io/projected/4f2f71c8-ba9b-43d4-8b14-6406ac50f2da-kube-api-access-fsdsd\") pod \"openshift-lws-operator-fd8ccff4c-xg82n\" (UID: \"4f2f71c8-ba9b-43d4-8b14-6406ac50f2da\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xg82n" Jul 07 20:06:37.566183 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:37.566108 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xg82n" Jul 07 20:06:37.695643 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:37.695507 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xg82n"] Jul 07 20:06:37.698412 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:06:37.698376 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f2f71c8_ba9b_43d4_8b14_6406ac50f2da.slice/crio-b03184d08a72081f2933133f6fb489359e7217606a46bcb790315819924ed890 WatchSource:0}: Error finding container b03184d08a72081f2933133f6fb489359e7217606a46bcb790315819924ed890: Status 404 returned error can't find the container with id b03184d08a72081f2933133f6fb489359e7217606a46bcb790315819924ed890 Jul 07 20:06:38.025053 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:38.025021 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xg82n" event={"ID":"4f2f71c8-ba9b-43d4-8b14-6406ac50f2da","Type":"ContainerStarted","Data":"b03184d08a72081f2933133f6fb489359e7217606a46bcb790315819924ed890"} Jul 07 20:06:41.040468 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:41.040288 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xg82n" event={"ID":"4f2f71c8-ba9b-43d4-8b14-6406ac50f2da","Type":"ContainerStarted","Data":"bcd0a9b701291d0191e641a2722b89e7baa39a5174d92c7af5adb92339309758"} Jul 07 20:06:41.061345 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:41.061291 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xg82n" podStartSLOduration=0.860688812 podStartE2EDuration="4.0612767s" totalImagesPullingTime="3.200587888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:06:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:06:41.05955245 +0000 UTC m=+321.214014400" watchObservedRunningTime="2026-07-07 20:06:41.0612767 +0000 UTC m=+321.215738649" Jul 07 20:06:57.528798 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:57.528763 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-659495cbcb-pw9cc"] Jul 07 20:06:57.535546 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:57.535526 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-659495cbcb-pw9cc" Jul 07 20:06:57.549047 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:57.549025 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 07 20:06:57.549315 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:57.549292 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zjd5h\"" Jul 07 20:06:57.549726 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:57.549711 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 20:06:57.550739 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:57.550720 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 20:06:57.551569 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:57.551552 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 20:06:57.568237 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:57.568213 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-659495cbcb-pw9cc"] Jul 07 20:06:57.695730 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:57.695698 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/510a6e1c-e835-4397-9f70-e1fe37af3f2b-tmp\") pod \"opendatahub-operator-controller-manager-659495cbcb-pw9cc\" (UID: \"510a6e1c-e835-4397-9f70-e1fe37af3f2b\") " pod="opendatahub/opendatahub-operator-controller-manager-659495cbcb-pw9cc" Jul 07 20:06:57.695886 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:57.695745 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/510a6e1c-e835-4397-9f70-e1fe37af3f2b-manifests\") pod \"opendatahub-operator-controller-manager-659495cbcb-pw9cc\" (UID: \"510a6e1c-e835-4397-9f70-e1fe37af3f2b\") " pod="opendatahub/opendatahub-operator-controller-manager-659495cbcb-pw9cc" Jul 07 20:06:57.695886 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:57.695764 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/510a6e1c-e835-4397-9f70-e1fe37af3f2b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-659495cbcb-pw9cc\" (UID: \"510a6e1c-e835-4397-9f70-e1fe37af3f2b\") " pod="opendatahub/opendatahub-operator-controller-manager-659495cbcb-pw9cc" Jul 07 20:06:57.695886 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:57.695847 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/510a6e1c-e835-4397-9f70-e1fe37af3f2b-webhook-cert\") pod \"opendatahub-operator-controller-manager-659495cbcb-pw9cc\" (UID: \"510a6e1c-e835-4397-9f70-e1fe37af3f2b\") " pod="opendatahub/opendatahub-operator-controller-manager-659495cbcb-pw9cc" Jul 07 20:06:57.695993 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:57.695892 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4lzt\" (UniqueName: \"kubernetes.io/projected/510a6e1c-e835-4397-9f70-e1fe37af3f2b-kube-api-access-z4lzt\") pod \"opendatahub-operator-controller-manager-659495cbcb-pw9cc\" (UID: \"510a6e1c-e835-4397-9f70-e1fe37af3f2b\") " pod="opendatahub/opendatahub-operator-controller-manager-659495cbcb-pw9cc" Jul 07 20:06:57.797197 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:57.797118 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/510a6e1c-e835-4397-9f70-e1fe37af3f2b-webhook-cert\") pod \"opendatahub-operator-controller-manager-659495cbcb-pw9cc\" (UID: \"510a6e1c-e835-4397-9f70-e1fe37af3f2b\") " pod="opendatahub/opendatahub-operator-controller-manager-659495cbcb-pw9cc" Jul 07 20:06:57.797341 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:57.797195 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4lzt\" (UniqueName: \"kubernetes.io/projected/510a6e1c-e835-4397-9f70-e1fe37af3f2b-kube-api-access-z4lzt\") pod \"opendatahub-operator-controller-manager-659495cbcb-pw9cc\" (UID: \"510a6e1c-e835-4397-9f70-e1fe37af3f2b\") " pod="opendatahub/opendatahub-operator-controller-manager-659495cbcb-pw9cc" Jul 07 20:06:57.797341 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:57.797253 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/510a6e1c-e835-4397-9f70-e1fe37af3f2b-tmp\") pod \"opendatahub-operator-controller-manager-659495cbcb-pw9cc\" (UID: \"510a6e1c-e835-4397-9f70-e1fe37af3f2b\") " pod="opendatahub/opendatahub-operator-controller-manager-659495cbcb-pw9cc" Jul 07 20:06:57.797493 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:57.797445 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/510a6e1c-e835-4397-9f70-e1fe37af3f2b-manifests\") pod \"opendatahub-operator-controller-manager-659495cbcb-pw9cc\" (UID: \"510a6e1c-e835-4397-9f70-e1fe37af3f2b\") " pod="opendatahub/opendatahub-operator-controller-manager-659495cbcb-pw9cc" Jul 07 20:06:57.797562 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:57.797489 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/510a6e1c-e835-4397-9f70-e1fe37af3f2b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-659495cbcb-pw9cc\" (UID: \"510a6e1c-e835-4397-9f70-e1fe37af3f2b\") " pod="opendatahub/opendatahub-operator-controller-manager-659495cbcb-pw9cc" Jul 07 20:06:57.797761 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:57.797741 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/510a6e1c-e835-4397-9f70-e1fe37af3f2b-manifests\") pod \"opendatahub-operator-controller-manager-659495cbcb-pw9cc\" (UID: \"510a6e1c-e835-4397-9f70-e1fe37af3f2b\") " pod="opendatahub/opendatahub-operator-controller-manager-659495cbcb-pw9cc" Jul 07 20:06:57.799582 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:57.799560 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/510a6e1c-e835-4397-9f70-e1fe37af3f2b-tmp\") pod \"opendatahub-operator-controller-manager-659495cbcb-pw9cc\" (UID: \"510a6e1c-e835-4397-9f70-e1fe37af3f2b\") " pod="opendatahub/opendatahub-operator-controller-manager-659495cbcb-pw9cc" Jul 07 20:06:57.799672 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:57.799658 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/510a6e1c-e835-4397-9f70-e1fe37af3f2b-webhook-cert\") pod \"opendatahub-operator-controller-manager-659495cbcb-pw9cc\" (UID: \"510a6e1c-e835-4397-9f70-e1fe37af3f2b\") " pod="opendatahub/opendatahub-operator-controller-manager-659495cbcb-pw9cc" Jul 07 20:06:57.799672 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:57.799665 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/510a6e1c-e835-4397-9f70-e1fe37af3f2b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-659495cbcb-pw9cc\" (UID: \"510a6e1c-e835-4397-9f70-e1fe37af3f2b\") " pod="opendatahub/opendatahub-operator-controller-manager-659495cbcb-pw9cc" Jul 07 20:06:57.805604 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:57.805581 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4lzt\" (UniqueName: \"kubernetes.io/projected/510a6e1c-e835-4397-9f70-e1fe37af3f2b-kube-api-access-z4lzt\") pod \"opendatahub-operator-controller-manager-659495cbcb-pw9cc\" (UID: \"510a6e1c-e835-4397-9f70-e1fe37af3f2b\") " pod="opendatahub/opendatahub-operator-controller-manager-659495cbcb-pw9cc" Jul 07 20:06:57.845569 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:57.845544 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-659495cbcb-pw9cc" Jul 07 20:06:57.984382 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:57.984349 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-659495cbcb-pw9cc"] Jul 07 20:06:57.987062 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:06:57.987038 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510a6e1c_e835_4397_9f70_e1fe37af3f2b.slice/crio-c1ea6958e0f9140b3296832742e5cf5bc15fd43bf16a48247b2ddaa4c54eff63 WatchSource:0}: Error finding container c1ea6958e0f9140b3296832742e5cf5bc15fd43bf16a48247b2ddaa4c54eff63: Status 404 returned error can't find the container with id c1ea6958e0f9140b3296832742e5cf5bc15fd43bf16a48247b2ddaa4c54eff63 Jul 07 20:06:58.114140 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:06:58.114081 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-659495cbcb-pw9cc" event={"ID":"510a6e1c-e835-4397-9f70-e1fe37af3f2b","Type":"ContainerStarted","Data":"c1ea6958e0f9140b3296832742e5cf5bc15fd43bf16a48247b2ddaa4c54eff63"} Jul 07 20:07:02.136485 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:02.136444 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-659495cbcb-pw9cc" event={"ID":"510a6e1c-e835-4397-9f70-e1fe37af3f2b","Type":"ContainerStarted","Data":"94a22a6fa097ce279bd4e587ea776f5f73834365022ca5ed9adcaad9c730787d"} Jul 07 20:07:03.141677 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:03.141641 2565 generic.go:356] "Generic (PLEG): container finished" podID="510a6e1c-e835-4397-9f70-e1fe37af3f2b" containerID="94a22a6fa097ce279bd4e587ea776f5f73834365022ca5ed9adcaad9c730787d" exitCode=0 Jul 07 20:07:03.142062 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:03.141735 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-659495cbcb-pw9cc" event={"ID":"510a6e1c-e835-4397-9f70-e1fe37af3f2b","Type":"ContainerDied","Data":"94a22a6fa097ce279bd4e587ea776f5f73834365022ca5ed9adcaad9c730787d"} Jul 07 20:07:04.153349 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:04.153319 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-659495cbcb-pw9cc" event={"ID":"510a6e1c-e835-4397-9f70-e1fe37af3f2b","Type":"ContainerStarted","Data":"b2f9250bbbab76b583b92d0a34eb2c7baa98b5e2441d8323a1ad0b12c12b7a65"} Jul 07 20:07:04.153725 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:04.153434 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-659495cbcb-pw9cc" Jul 07 20:07:04.178702 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:04.178650 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-659495cbcb-pw9cc" podStartSLOduration=2.521988262 podStartE2EDuration="7.178636863s" totalImagesPullingTime="3.584985552s" totalInitContainerRuntime="1.071663049s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:06:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:07:04.177106024 +0000 UTC m=+344.331567973" watchObservedRunningTime="2026-07-07 20:07:04.178636863 +0000 UTC m=+344.333098812" Jul 07 20:07:15.160772 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:15.160696 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-659495cbcb-pw9cc" Jul 07 20:07:16.384125 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:16.384089 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-699bddd6bc-z9qtq"] Jul 07 20:07:16.387193 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:16.387171 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-699bddd6bc-z9qtq" Jul 07 20:07:16.389607 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:16.389584 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-fjfv5\"" Jul 07 20:07:16.390726 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:16.390700 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 07 20:07:16.390867 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:16.390726 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 07 20:07:16.390867 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:16.390702 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 07 20:07:16.397934 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:16.397912 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-699bddd6bc-z9qtq"] Jul 07 20:07:16.456371 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:16.456341 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4ebd5d73-cc74-4811-9393-4c64df8de9e0-manager-config\") pod \"lws-controller-manager-699bddd6bc-z9qtq\" (UID: \"4ebd5d73-cc74-4811-9393-4c64df8de9e0\") " pod="openshift-lws-operator/lws-controller-manager-699bddd6bc-z9qtq" Jul 07 20:07:16.456529 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:16.456376 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ebd5d73-cc74-4811-9393-4c64df8de9e0-metrics-cert\") pod \"lws-controller-manager-699bddd6bc-z9qtq\" (UID: \"4ebd5d73-cc74-4811-9393-4c64df8de9e0\") " pod="openshift-lws-operator/lws-controller-manager-699bddd6bc-z9qtq" Jul 07 20:07:16.456529 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:16.456437 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bfgl\" (UniqueName: \"kubernetes.io/projected/4ebd5d73-cc74-4811-9393-4c64df8de9e0-kube-api-access-9bfgl\") pod \"lws-controller-manager-699bddd6bc-z9qtq\" (UID: \"4ebd5d73-cc74-4811-9393-4c64df8de9e0\") " pod="openshift-lws-operator/lws-controller-manager-699bddd6bc-z9qtq" Jul 07 20:07:16.456529 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:16.456462 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ebd5d73-cc74-4811-9393-4c64df8de9e0-cert\") pod \"lws-controller-manager-699bddd6bc-z9qtq\" (UID: \"4ebd5d73-cc74-4811-9393-4c64df8de9e0\") " pod="openshift-lws-operator/lws-controller-manager-699bddd6bc-z9qtq" Jul 07 20:07:16.557736 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:16.557701 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4ebd5d73-cc74-4811-9393-4c64df8de9e0-manager-config\") pod \"lws-controller-manager-699bddd6bc-z9qtq\" (UID: \"4ebd5d73-cc74-4811-9393-4c64df8de9e0\") " pod="openshift-lws-operator/lws-controller-manager-699bddd6bc-z9qtq" Jul 07 20:07:16.557881 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:16.557743 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ebd5d73-cc74-4811-9393-4c64df8de9e0-metrics-cert\") pod \"lws-controller-manager-699bddd6bc-z9qtq\" (UID: \"4ebd5d73-cc74-4811-9393-4c64df8de9e0\") " pod="openshift-lws-operator/lws-controller-manager-699bddd6bc-z9qtq" Jul 07 20:07:16.557881 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:16.557790 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bfgl\" (UniqueName: \"kubernetes.io/projected/4ebd5d73-cc74-4811-9393-4c64df8de9e0-kube-api-access-9bfgl\") pod \"lws-controller-manager-699bddd6bc-z9qtq\" (UID: \"4ebd5d73-cc74-4811-9393-4c64df8de9e0\") " pod="openshift-lws-operator/lws-controller-manager-699bddd6bc-z9qtq" Jul 07 20:07:16.557881 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:16.557835 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ebd5d73-cc74-4811-9393-4c64df8de9e0-cert\") pod \"lws-controller-manager-699bddd6bc-z9qtq\" (UID: \"4ebd5d73-cc74-4811-9393-4c64df8de9e0\") " pod="openshift-lws-operator/lws-controller-manager-699bddd6bc-z9qtq" Jul 07 20:07:16.558346 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:16.558326 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4ebd5d73-cc74-4811-9393-4c64df8de9e0-manager-config\") pod \"lws-controller-manager-699bddd6bc-z9qtq\" (UID: \"4ebd5d73-cc74-4811-9393-4c64df8de9e0\") " pod="openshift-lws-operator/lws-controller-manager-699bddd6bc-z9qtq" Jul 07 20:07:16.560248 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:16.560226 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ebd5d73-cc74-4811-9393-4c64df8de9e0-cert\") pod \"lws-controller-manager-699bddd6bc-z9qtq\" (UID: \"4ebd5d73-cc74-4811-9393-4c64df8de9e0\") " pod="openshift-lws-operator/lws-controller-manager-699bddd6bc-z9qtq" Jul 07 20:07:16.560325 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:16.560273 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ebd5d73-cc74-4811-9393-4c64df8de9e0-metrics-cert\") pod \"lws-controller-manager-699bddd6bc-z9qtq\" (UID: \"4ebd5d73-cc74-4811-9393-4c64df8de9e0\") " pod="openshift-lws-operator/lws-controller-manager-699bddd6bc-z9qtq" Jul 07 20:07:16.568886 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:16.568866 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bfgl\" (UniqueName: \"kubernetes.io/projected/4ebd5d73-cc74-4811-9393-4c64df8de9e0-kube-api-access-9bfgl\") pod \"lws-controller-manager-699bddd6bc-z9qtq\" (UID: \"4ebd5d73-cc74-4811-9393-4c64df8de9e0\") " pod="openshift-lws-operator/lws-controller-manager-699bddd6bc-z9qtq" Jul 07 20:07:16.696435 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:16.696363 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-699bddd6bc-z9qtq" Jul 07 20:07:16.849892 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:16.849860 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-699bddd6bc-z9qtq"] Jul 07 20:07:16.851178 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:07:16.851150 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ebd5d73_cc74_4811_9393_4c64df8de9e0.slice/crio-4a87b47d5accd7b5352b26e3462cd72daf8a760235135e037bce46a13540dfa2 WatchSource:0}: Error finding container 4a87b47d5accd7b5352b26e3462cd72daf8a760235135e037bce46a13540dfa2: Status 404 returned error can't find the container with id 4a87b47d5accd7b5352b26e3462cd72daf8a760235135e037bce46a13540dfa2 Jul 07 20:07:17.210726 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:17.210690 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-699bddd6bc-z9qtq" event={"ID":"4ebd5d73-cc74-4811-9393-4c64df8de9e0","Type":"ContainerStarted","Data":"4a87b47d5accd7b5352b26e3462cd72daf8a760235135e037bce46a13540dfa2"} Jul 07 20:07:17.453527 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:17.453496 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-68f4c445fc-48crl"] Jul 07 20:07:17.457644 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:17.457626 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68f4c445fc-48crl" Jul 07 20:07:17.465049 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:17.465028 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 07 20:07:17.465373 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:17.465355 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 07 20:07:17.466662 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:17.466644 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9jc7\" (UniqueName: \"kubernetes.io/projected/18992f4a-342c-4bd6-81fb-5e5fc9d951d1-kube-api-access-x9jc7\") pod \"kube-auth-proxy-68f4c445fc-48crl\" (UID: \"18992f4a-342c-4bd6-81fb-5e5fc9d951d1\") " pod="openshift-ingress/kube-auth-proxy-68f4c445fc-48crl" Jul 07 20:07:17.466742 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:17.466690 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18992f4a-342c-4bd6-81fb-5e5fc9d951d1-tmp\") pod \"kube-auth-proxy-68f4c445fc-48crl\" (UID: \"18992f4a-342c-4bd6-81fb-5e5fc9d951d1\") " pod="openshift-ingress/kube-auth-proxy-68f4c445fc-48crl" Jul 07 20:07:17.466742 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:17.466736 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18992f4a-342c-4bd6-81fb-5e5fc9d951d1-tls-certs\") pod \"kube-auth-proxy-68f4c445fc-48crl\" (UID: \"18992f4a-342c-4bd6-81fb-5e5fc9d951d1\") " pod="openshift-ingress/kube-auth-proxy-68f4c445fc-48crl" Jul 07 20:07:17.470395 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:17.470375 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 20:07:17.474099 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:17.474087 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-wqmc7\"" Jul 07 20:07:17.478816 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:17.478797 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 20:07:17.491650 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:17.491631 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68f4c445fc-48crl"] Jul 07 20:07:17.567450 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:17.567424 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9jc7\" (UniqueName: \"kubernetes.io/projected/18992f4a-342c-4bd6-81fb-5e5fc9d951d1-kube-api-access-x9jc7\") pod \"kube-auth-proxy-68f4c445fc-48crl\" (UID: \"18992f4a-342c-4bd6-81fb-5e5fc9d951d1\") " pod="openshift-ingress/kube-auth-proxy-68f4c445fc-48crl" Jul 07 20:07:17.567557 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:17.567484 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18992f4a-342c-4bd6-81fb-5e5fc9d951d1-tmp\") pod \"kube-auth-proxy-68f4c445fc-48crl\" (UID: \"18992f4a-342c-4bd6-81fb-5e5fc9d951d1\") " pod="openshift-ingress/kube-auth-proxy-68f4c445fc-48crl" Jul 07 20:07:17.567557 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:17.567526 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18992f4a-342c-4bd6-81fb-5e5fc9d951d1-tls-certs\") pod \"kube-auth-proxy-68f4c445fc-48crl\" (UID: \"18992f4a-342c-4bd6-81fb-5e5fc9d951d1\") " pod="openshift-ingress/kube-auth-proxy-68f4c445fc-48crl" Jul 07 20:07:17.569744 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:17.569722 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18992f4a-342c-4bd6-81fb-5e5fc9d951d1-tmp\") pod \"kube-auth-proxy-68f4c445fc-48crl\" (UID: \"18992f4a-342c-4bd6-81fb-5e5fc9d951d1\") " pod="openshift-ingress/kube-auth-proxy-68f4c445fc-48crl" Jul 07 20:07:17.569901 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:17.569842 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18992f4a-342c-4bd6-81fb-5e5fc9d951d1-tls-certs\") pod \"kube-auth-proxy-68f4c445fc-48crl\" (UID: \"18992f4a-342c-4bd6-81fb-5e5fc9d951d1\") " pod="openshift-ingress/kube-auth-proxy-68f4c445fc-48crl" Jul 07 20:07:17.582758 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:17.582738 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9jc7\" (UniqueName: \"kubernetes.io/projected/18992f4a-342c-4bd6-81fb-5e5fc9d951d1-kube-api-access-x9jc7\") pod \"kube-auth-proxy-68f4c445fc-48crl\" (UID: \"18992f4a-342c-4bd6-81fb-5e5fc9d951d1\") " pod="openshift-ingress/kube-auth-proxy-68f4c445fc-48crl" Jul 07 20:07:17.767574 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:17.767493 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68f4c445fc-48crl" Jul 07 20:07:17.959507 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:17.959470 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68f4c445fc-48crl"] Jul 07 20:07:17.963419 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:07:17.963374 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18992f4a_342c_4bd6_81fb_5e5fc9d951d1.slice/crio-82d69c5748358542cd79d64c0868ad671b7cb1ece47999d2fb8d0f4d4fdce2b5 WatchSource:0}: Error finding container 82d69c5748358542cd79d64c0868ad671b7cb1ece47999d2fb8d0f4d4fdce2b5: Status 404 returned error can't find the container with id 82d69c5748358542cd79d64c0868ad671b7cb1ece47999d2fb8d0f4d4fdce2b5 Jul 07 20:07:18.216861 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:18.216830 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68f4c445fc-48crl" event={"ID":"18992f4a-342c-4bd6-81fb-5e5fc9d951d1","Type":"ContainerStarted","Data":"82d69c5748358542cd79d64c0868ad671b7cb1ece47999d2fb8d0f4d4fdce2b5"} Jul 07 20:07:19.223166 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:19.223129 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-699bddd6bc-z9qtq" event={"ID":"4ebd5d73-cc74-4811-9393-4c64df8de9e0","Type":"ContainerStarted","Data":"b9ebc9edab65e7f1387e8b43256344c815f2eb5c2ec841661a509e68ea74d159"} Jul 07 20:07:19.223649 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:19.223287 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-699bddd6bc-z9qtq" Jul 07 20:07:19.252208 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:19.252163 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-699bddd6bc-z9qtq" podStartSLOduration=1.635189045 podStartE2EDuration="3.252148596s" totalImagesPullingTime="1.616959551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:07:19.249844571 +0000 UTC m=+359.404306521" watchObservedRunningTime="2026-07-07 20:07:19.252148596 +0000 UTC m=+359.406610544" Jul 07 20:07:21.952781 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:21.952760 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 07 20:07:22.238962 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:22.238879 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68f4c445fc-48crl" event={"ID":"18992f4a-342c-4bd6-81fb-5e5fc9d951d1","Type":"ContainerStarted","Data":"a540fd3d39cef5b1bfb7c4765fd386a1af1e1dde2ea1d45cf9fe86d50b138d2e"} Jul 07 20:07:22.259192 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:22.259130 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-68f4c445fc-48crl" podStartSLOduration=1.2771986929999999 podStartE2EDuration="5.259115263s" totalImagesPullingTime="3.98191657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:07:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:07:22.258353561 +0000 UTC m=+362.412815508" watchObservedRunningTime="2026-07-07 20:07:22.259115263 +0000 UTC m=+362.413577211" Jul 07 20:07:30.229751 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:07:30.229717 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-699bddd6bc-z9qtq" Jul 07 20:09:12.430729 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:12.430699 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-h7wf9"] Jul 07 20:09:12.433253 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:12.433236 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h7wf9" Jul 07 20:09:12.435935 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:12.435910 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 07 20:09:12.436039 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:12.435922 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 07 20:09:12.436039 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:12.435922 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 20:09:12.436147 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:12.435976 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 20:09:12.437064 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:12.437046 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zqrbq\"" Jul 07 20:09:12.449501 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:12.449482 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-h7wf9"] Jul 07 20:09:12.577697 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:12.577664 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/83bb2bf4-0574-43a0-80a2-43560f286fc7-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-h7wf9\" (UID: \"83bb2bf4-0574-43a0-80a2-43560f286fc7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h7wf9" Jul 07 20:09:12.577867 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:12.577714 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/83bb2bf4-0574-43a0-80a2-43560f286fc7-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-h7wf9\" (UID: \"83bb2bf4-0574-43a0-80a2-43560f286fc7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h7wf9" Jul 07 20:09:12.577867 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:12.577786 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqsd2\" (UniqueName: \"kubernetes.io/projected/83bb2bf4-0574-43a0-80a2-43560f286fc7-kube-api-access-mqsd2\") pod \"kuadrant-console-plugin-66b5774985-h7wf9\" (UID: \"83bb2bf4-0574-43a0-80a2-43560f286fc7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h7wf9" Jul 07 20:09:12.679004 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:12.678942 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqsd2\" (UniqueName: \"kubernetes.io/projected/83bb2bf4-0574-43a0-80a2-43560f286fc7-kube-api-access-mqsd2\") pod \"kuadrant-console-plugin-66b5774985-h7wf9\" (UID: \"83bb2bf4-0574-43a0-80a2-43560f286fc7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h7wf9" Jul 07 20:09:12.679183 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:12.679107 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/83bb2bf4-0574-43a0-80a2-43560f286fc7-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-h7wf9\" (UID: \"83bb2bf4-0574-43a0-80a2-43560f286fc7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h7wf9" Jul 07 20:09:12.679183 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:12.679179 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/83bb2bf4-0574-43a0-80a2-43560f286fc7-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-h7wf9\" (UID: \"83bb2bf4-0574-43a0-80a2-43560f286fc7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h7wf9" Jul 07 20:09:12.680042 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:12.680011 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/83bb2bf4-0574-43a0-80a2-43560f286fc7-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-h7wf9\" (UID: \"83bb2bf4-0574-43a0-80a2-43560f286fc7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h7wf9" Jul 07 20:09:12.682361 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:12.682299 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/83bb2bf4-0574-43a0-80a2-43560f286fc7-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-h7wf9\" (UID: \"83bb2bf4-0574-43a0-80a2-43560f286fc7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h7wf9" Jul 07 20:09:12.688550 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:12.688523 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqsd2\" (UniqueName: \"kubernetes.io/projected/83bb2bf4-0574-43a0-80a2-43560f286fc7-kube-api-access-mqsd2\") pod \"kuadrant-console-plugin-66b5774985-h7wf9\" (UID: \"83bb2bf4-0574-43a0-80a2-43560f286fc7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h7wf9" Jul 07 20:09:12.742374 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:12.742349 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h7wf9" Jul 07 20:09:12.879163 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:12.879127 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-h7wf9"] Jul 07 20:09:12.883787 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:09:12.883760 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83bb2bf4_0574_43a0_80a2_43560f286fc7.slice/crio-f86763504abed6fd05d704999ae1a44e1063ad0fccdf562c9c165dd4ece4d529 WatchSource:0}: Error finding container f86763504abed6fd05d704999ae1a44e1063ad0fccdf562c9c165dd4ece4d529: Status 404 returned error can't find the container with id f86763504abed6fd05d704999ae1a44e1063ad0fccdf562c9c165dd4ece4d529 Jul 07 20:09:13.732414 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:13.732369 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h7wf9" event={"ID":"83bb2bf4-0574-43a0-80a2-43560f286fc7","Type":"ContainerStarted","Data":"f86763504abed6fd05d704999ae1a44e1063ad0fccdf562c9c165dd4ece4d529"} Jul 07 20:09:23.139774 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:23.139706 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j69cz"] Jul 07 20:09:23.143628 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:23.143602 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j69cz" Jul 07 20:09:23.146232 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:23.146206 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-vvksh\"" Jul 07 20:09:23.155480 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:23.155454 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j69cz"] Jul 07 20:09:23.281374 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:23.281332 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jdvp\" (UniqueName: \"kubernetes.io/projected/e1085214-ede2-463c-abd1-74a74cd10cb0-kube-api-access-4jdvp\") pod \"limitador-operator-controller-manager-859d7b78bd-j69cz\" (UID: \"e1085214-ede2-463c-abd1-74a74cd10cb0\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j69cz" Jul 07 20:09:23.382646 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:23.382617 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jdvp\" (UniqueName: \"kubernetes.io/projected/e1085214-ede2-463c-abd1-74a74cd10cb0-kube-api-access-4jdvp\") pod \"limitador-operator-controller-manager-859d7b78bd-j69cz\" (UID: \"e1085214-ede2-463c-abd1-74a74cd10cb0\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j69cz" Jul 07 20:09:23.394181 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:23.394119 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jdvp\" (UniqueName: \"kubernetes.io/projected/e1085214-ede2-463c-abd1-74a74cd10cb0-kube-api-access-4jdvp\") pod \"limitador-operator-controller-manager-859d7b78bd-j69cz\" (UID: \"e1085214-ede2-463c-abd1-74a74cd10cb0\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j69cz" Jul 07 20:09:23.458081 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:23.458028 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j69cz" Jul 07 20:09:23.602765 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:23.602741 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j69cz"] Jul 07 20:09:23.604967 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:09:23.604934 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1085214_ede2_463c_abd1_74a74cd10cb0.slice/crio-55eae04b6777bfbae1843856def2f3391bf5ac393f5ab367539c844f53d87de8 WatchSource:0}: Error finding container 55eae04b6777bfbae1843856def2f3391bf5ac393f5ab367539c844f53d87de8: Status 404 returned error can't find the container with id 55eae04b6777bfbae1843856def2f3391bf5ac393f5ab367539c844f53d87de8 Jul 07 20:09:23.784042 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:23.783957 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j69cz" event={"ID":"e1085214-ede2-463c-abd1-74a74cd10cb0","Type":"ContainerStarted","Data":"55eae04b6777bfbae1843856def2f3391bf5ac393f5ab367539c844f53d87de8"} Jul 07 20:09:40.875995 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:40.875958 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h7wf9" event={"ID":"83bb2bf4-0574-43a0-80a2-43560f286fc7","Type":"ContainerStarted","Data":"31b7371582749df423df64eb6cb48a185ecae9b6d589c1e1ad204b10e9f25226"} Jul 07 20:09:40.877552 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:40.877529 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j69cz" event={"ID":"e1085214-ede2-463c-abd1-74a74cd10cb0","Type":"ContainerStarted","Data":"b4bc21f55bfa63adcef44b15060c6647332f6daeaad674fb6d315db5e2e205ee"} Jul 07 20:09:40.877662 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:40.877591 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j69cz" Jul 07 20:09:40.892198 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:40.892148 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-h7wf9" podStartSLOduration=2.451066755 podStartE2EDuration="28.892134439s" totalImagesPullingTime="26.441067684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:09:40.891589331 +0000 UTC m=+501.046051281" watchObservedRunningTime="2026-07-07 20:09:40.892134439 +0000 UTC m=+501.046596388" Jul 07 20:09:40.907139 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:40.907098 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j69cz" podStartSLOduration=1.597995039 podStartE2EDuration="17.907085818s" totalImagesPullingTime="16.309090779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:09:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:09:40.906188606 +0000 UTC m=+501.060650555" watchObservedRunningTime="2026-07-07 20:09:40.907085818 +0000 UTC m=+501.061547767" Jul 07 20:09:51.884267 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:09:51.884237 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-j69cz" Jul 07 20:11:19.718239 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:19.718208 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-77476c4d87-twlcf"] Jul 07 20:11:19.720587 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:19.720568 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-77476c4d87-twlcf" Jul 07 20:11:19.723533 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:19.723512 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 07 20:11:19.723637 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:19.723516 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-87gvt\"" Jul 07 20:11:19.732470 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:19.732449 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-77476c4d87-twlcf"] Jul 07 20:11:19.832040 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:19.832011 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce7010ee-0687-4868-9077-7dfdd7a3545e-maas-api-tls\") pod \"maas-api-77476c4d87-twlcf\" (UID: \"ce7010ee-0687-4868-9077-7dfdd7a3545e\") " pod="opendatahub/maas-api-77476c4d87-twlcf" Jul 07 20:11:19.832193 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:19.832152 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce7010ee-0687-4868-9077-7dfdd7a3545e-openshift-service-ca\") pod \"maas-api-77476c4d87-twlcf\" (UID: \"ce7010ee-0687-4868-9077-7dfdd7a3545e\") " pod="opendatahub/maas-api-77476c4d87-twlcf" Jul 07 20:11:19.832377 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:19.832353 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn42b\" (UniqueName: \"kubernetes.io/projected/ce7010ee-0687-4868-9077-7dfdd7a3545e-kube-api-access-jn42b\") pod \"maas-api-77476c4d87-twlcf\" (UID: \"ce7010ee-0687-4868-9077-7dfdd7a3545e\") " pod="opendatahub/maas-api-77476c4d87-twlcf" Jul 07 20:11:19.933527 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:19.933497 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce7010ee-0687-4868-9077-7dfdd7a3545e-maas-api-tls\") pod \"maas-api-77476c4d87-twlcf\" (UID: \"ce7010ee-0687-4868-9077-7dfdd7a3545e\") " pod="opendatahub/maas-api-77476c4d87-twlcf" Jul 07 20:11:19.933702 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:19.933606 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce7010ee-0687-4868-9077-7dfdd7a3545e-openshift-service-ca\") pod \"maas-api-77476c4d87-twlcf\" (UID: \"ce7010ee-0687-4868-9077-7dfdd7a3545e\") " pod="opendatahub/maas-api-77476c4d87-twlcf" Jul 07 20:11:19.933702 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:11:19.933672 2565 secret.go:189] Couldn't get secret opendatahub/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 07 20:11:19.933836 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:19.933704 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn42b\" (UniqueName: \"kubernetes.io/projected/ce7010ee-0687-4868-9077-7dfdd7a3545e-kube-api-access-jn42b\") pod \"maas-api-77476c4d87-twlcf\" (UID: \"ce7010ee-0687-4868-9077-7dfdd7a3545e\") " pod="opendatahub/maas-api-77476c4d87-twlcf" Jul 07 20:11:19.933836 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:11:19.933742 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce7010ee-0687-4868-9077-7dfdd7a3545e-maas-api-tls podName:ce7010ee-0687-4868-9077-7dfdd7a3545e nodeName:}" failed. No retries permitted until 2026-07-07 20:11:20.43371947 +0000 UTC m=+600.588181398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ce7010ee-0687-4868-9077-7dfdd7a3545e-maas-api-tls") pod "maas-api-77476c4d87-twlcf" (UID: "ce7010ee-0687-4868-9077-7dfdd7a3545e") : secret "maas-api-serving-cert" not found Jul 07 20:11:19.934418 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:19.934370 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce7010ee-0687-4868-9077-7dfdd7a3545e-openshift-service-ca\") pod \"maas-api-77476c4d87-twlcf\" (UID: \"ce7010ee-0687-4868-9077-7dfdd7a3545e\") " pod="opendatahub/maas-api-77476c4d87-twlcf" Jul 07 20:11:19.944071 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:19.944045 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn42b\" (UniqueName: \"kubernetes.io/projected/ce7010ee-0687-4868-9077-7dfdd7a3545e-kube-api-access-jn42b\") pod \"maas-api-77476c4d87-twlcf\" (UID: \"ce7010ee-0687-4868-9077-7dfdd7a3545e\") " pod="opendatahub/maas-api-77476c4d87-twlcf" Jul 07 20:11:20.436765 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:20.436730 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce7010ee-0687-4868-9077-7dfdd7a3545e-maas-api-tls\") pod \"maas-api-77476c4d87-twlcf\" (UID: \"ce7010ee-0687-4868-9077-7dfdd7a3545e\") " pod="opendatahub/maas-api-77476c4d87-twlcf" Jul 07 20:11:20.439085 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:20.439056 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce7010ee-0687-4868-9077-7dfdd7a3545e-maas-api-tls\") pod \"maas-api-77476c4d87-twlcf\" (UID: \"ce7010ee-0687-4868-9077-7dfdd7a3545e\") " pod="opendatahub/maas-api-77476c4d87-twlcf" Jul 07 20:11:20.630655 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:20.630609 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-77476c4d87-twlcf" Jul 07 20:11:20.761440 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:20.761368 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-77476c4d87-twlcf"] Jul 07 20:11:20.764081 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:11:20.764056 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce7010ee_0687_4868_9077_7dfdd7a3545e.slice/crio-29d5efb054347edac961872eaa40d50631f7c7c449a329d699ec0f5dc544770b WatchSource:0}: Error finding container 29d5efb054347edac961872eaa40d50631f7c7c449a329d699ec0f5dc544770b: Status 404 returned error can't find the container with id 29d5efb054347edac961872eaa40d50631f7c7c449a329d699ec0f5dc544770b Jul 07 20:11:21.335876 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:21.335835 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-77476c4d87-twlcf" event={"ID":"ce7010ee-0687-4868-9077-7dfdd7a3545e","Type":"ContainerStarted","Data":"29d5efb054347edac961872eaa40d50631f7c7c449a329d699ec0f5dc544770b"} Jul 07 20:11:23.346100 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:23.346063 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-77476c4d87-twlcf" event={"ID":"ce7010ee-0687-4868-9077-7dfdd7a3545e","Type":"ContainerStarted","Data":"5203fb67bdab7f8dfe236e05bb8c9199b2f0655e014d682dca5366854fc99560"} Jul 07 20:11:23.346472 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:23.346197 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-77476c4d87-twlcf" Jul 07 20:11:23.366265 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:23.366218 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-77476c4d87-twlcf" podStartSLOduration=2.322487457 podStartE2EDuration="4.366202909s" totalImagesPullingTime="2.043715452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:11:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:11:23.364860287 +0000 UTC m=+603.519322236" watchObservedRunningTime="2026-07-07 20:11:23.366202909 +0000 UTC m=+603.520664858" Jul 07 20:11:29.356506 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:29.356476 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-77476c4d87-twlcf" Jul 07 20:11:42.060556 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:42.060464 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2"] Jul 07 20:11:42.063075 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:42.063058 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2" Jul 07 20:11:42.066121 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:42.066099 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 07 20:11:42.066240 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:42.066108 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 07 20:11:42.066240 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:42.066163 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 07 20:11:42.067189 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:42.067173 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8k5vx\"" Jul 07 20:11:42.075704 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:42.075684 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2"] Jul 07 20:11:42.132376 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:42.132344 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14085b5e-a02b-43d3-b966-4f777cf80c2c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2\" (UID: \"14085b5e-a02b-43d3-b966-4f777cf80c2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2" Jul 07 20:11:42.132538 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:42.132391 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14085b5e-a02b-43d3-b966-4f777cf80c2c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2\" (UID: \"14085b5e-a02b-43d3-b966-4f777cf80c2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2" Jul 07 20:11:42.132538 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:42.132459 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14085b5e-a02b-43d3-b966-4f777cf80c2c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2\" (UID: \"14085b5e-a02b-43d3-b966-4f777cf80c2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2" Jul 07 20:11:42.132538 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:42.132499 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14085b5e-a02b-43d3-b966-4f777cf80c2c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2\" (UID: \"14085b5e-a02b-43d3-b966-4f777cf80c2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2" Jul 07 20:11:42.132707 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:42.132540 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwj67\" (UniqueName: \"kubernetes.io/projected/14085b5e-a02b-43d3-b966-4f777cf80c2c-kube-api-access-jwj67\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2\" (UID: \"14085b5e-a02b-43d3-b966-4f777cf80c2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2" Jul 07 20:11:42.132707 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:42.132563 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14085b5e-a02b-43d3-b966-4f777cf80c2c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2\" (UID: \"14085b5e-a02b-43d3-b966-4f777cf80c2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2" Jul 07 20:11:42.233704 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:42.233663 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14085b5e-a02b-43d3-b966-4f777cf80c2c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2\" (UID: \"14085b5e-a02b-43d3-b966-4f777cf80c2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2" Jul 07 20:11:42.233866 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:42.233713 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14085b5e-a02b-43d3-b966-4f777cf80c2c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2\" (UID: \"14085b5e-a02b-43d3-b966-4f777cf80c2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2" Jul 07 20:11:42.233866 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:42.233765 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14085b5e-a02b-43d3-b966-4f777cf80c2c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2\" (UID: \"14085b5e-a02b-43d3-b966-4f777cf80c2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2" Jul 07 20:11:42.233866 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:42.233816 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwj67\" (UniqueName: \"kubernetes.io/projected/14085b5e-a02b-43d3-b966-4f777cf80c2c-kube-api-access-jwj67\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2\" (UID: \"14085b5e-a02b-43d3-b966-4f777cf80c2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2" Jul 07 20:11:42.233866 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:42.233839 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14085b5e-a02b-43d3-b966-4f777cf80c2c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2\" (UID: \"14085b5e-a02b-43d3-b966-4f777cf80c2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2" Jul 07 20:11:42.234012 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:42.233986 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14085b5e-a02b-43d3-b966-4f777cf80c2c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2\" (UID: \"14085b5e-a02b-43d3-b966-4f777cf80c2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2" Jul 07 20:11:42.234160 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:42.234141 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14085b5e-a02b-43d3-b966-4f777cf80c2c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2\" (UID: \"14085b5e-a02b-43d3-b966-4f777cf80c2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2" Jul 07 20:11:42.234233 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:42.234177 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14085b5e-a02b-43d3-b966-4f777cf80c2c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2\" (UID: \"14085b5e-a02b-43d3-b966-4f777cf80c2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2" Jul 07 20:11:42.234342 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:42.234320 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14085b5e-a02b-43d3-b966-4f777cf80c2c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2\" (UID: \"14085b5e-a02b-43d3-b966-4f777cf80c2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2" Jul 07 20:11:42.236148 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:42.236063 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14085b5e-a02b-43d3-b966-4f777cf80c2c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2\" (UID: \"14085b5e-a02b-43d3-b966-4f777cf80c2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2" Jul 07 20:11:42.236295 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:42.236274 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14085b5e-a02b-43d3-b966-4f777cf80c2c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2\" (UID: \"14085b5e-a02b-43d3-b966-4f777cf80c2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2" Jul 07 20:11:42.242720 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:42.242701 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwj67\" (UniqueName: \"kubernetes.io/projected/14085b5e-a02b-43d3-b966-4f777cf80c2c-kube-api-access-jwj67\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2\" (UID: \"14085b5e-a02b-43d3-b966-4f777cf80c2c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2" Jul 07 20:11:42.373207 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:42.373115 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2" Jul 07 20:11:42.505484 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:42.505462 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2"] Jul 07 20:11:42.507489 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:11:42.507461 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14085b5e_a02b_43d3_b966_4f777cf80c2c.slice/crio-c1d3a060e7ad304b4b7ae8e8753df2d67fd763f2d1a9dbcbdaa6f9d32ad37b24 WatchSource:0}: Error finding container c1d3a060e7ad304b4b7ae8e8753df2d67fd763f2d1a9dbcbdaa6f9d32ad37b24: Status 404 returned error can't find the container with id c1d3a060e7ad304b4b7ae8e8753df2d67fd763f2d1a9dbcbdaa6f9d32ad37b24 Jul 07 20:11:42.508785 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:42.508765 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 20:11:43.441897 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:43.441849 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2" event={"ID":"14085b5e-a02b-43d3-b966-4f777cf80c2c","Type":"ContainerStarted","Data":"c1d3a060e7ad304b4b7ae8e8753df2d67fd763f2d1a9dbcbdaa6f9d32ad37b24"} Jul 07 20:11:44.447350 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:44.447316 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2" event={"ID":"14085b5e-a02b-43d3-b966-4f777cf80c2c","Type":"ContainerStarted","Data":"2e7158207cad4a8e865d14bf6d176a53305d2323ef3e1b3163bfb8deac5521c1"} Jul 07 20:11:44.466723 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:44.466672 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2" podStartSLOduration=1.302615611 podStartE2EDuration="2.466656375s" totalImagesPullingTime="1.164040764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:11:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:11:44.464785924 +0000 UTC m=+624.619247867" watchObservedRunningTime="2026-07-07 20:11:44.466656375 +0000 UTC m=+624.621118323" Jul 07 20:11:52.373840 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:52.373803 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2" Jul 07 20:11:52.374235 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:52.373863 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2" Jul 07 20:11:52.386487 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:52.386467 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2" Jul 07 20:11:52.502122 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:11:52.502099 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2" Jul 07 20:12:09.554841 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:12:09.554801 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w"] Jul 07 20:12:09.559306 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:12:09.559284 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w" Jul 07 20:12:09.561723 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:12:09.561688 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 07 20:12:09.568083 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:12:09.568059 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w"] Jul 07 20:12:09.592299 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:12:09.592261 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47b97\" (UniqueName: \"kubernetes.io/projected/c27ba098-46ba-4167-bedd-da6f0a16085e-kube-api-access-47b97\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w\" (UID: \"c27ba098-46ba-4167-bedd-da6f0a16085e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w" Jul 07 20:12:09.592465 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:12:09.592362 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c27ba098-46ba-4167-bedd-da6f0a16085e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w\" (UID: \"c27ba098-46ba-4167-bedd-da6f0a16085e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w" Jul 07 20:12:09.592465 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:12:09.592419 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c27ba098-46ba-4167-bedd-da6f0a16085e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w\" (UID: \"c27ba098-46ba-4167-bedd-da6f0a16085e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w" Jul 07 20:12:09.592556 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:12:09.592501 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c27ba098-46ba-4167-bedd-da6f0a16085e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w\" (UID: \"c27ba098-46ba-4167-bedd-da6f0a16085e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w" Jul 07 20:12:09.592556 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:12:09.592539 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c27ba098-46ba-4167-bedd-da6f0a16085e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w\" (UID: \"c27ba098-46ba-4167-bedd-da6f0a16085e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w" Jul 07 20:12:09.592628 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:12:09.592565 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27ba098-46ba-4167-bedd-da6f0a16085e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w\" (UID: \"c27ba098-46ba-4167-bedd-da6f0a16085e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w" Jul 07 20:12:09.693478 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:12:09.693448 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c27ba098-46ba-4167-bedd-da6f0a16085e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w\" (UID: \"c27ba098-46ba-4167-bedd-da6f0a16085e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w" Jul 07 20:12:09.693622 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:12:09.693484 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c27ba098-46ba-4167-bedd-da6f0a16085e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w\" (UID: \"c27ba098-46ba-4167-bedd-da6f0a16085e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w" Jul 07 20:12:09.693622 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:12:09.693502 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27ba098-46ba-4167-bedd-da6f0a16085e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w\" (UID: \"c27ba098-46ba-4167-bedd-da6f0a16085e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w" Jul 07 20:12:09.693622 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:12:09.693539 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47b97\" (UniqueName: \"kubernetes.io/projected/c27ba098-46ba-4167-bedd-da6f0a16085e-kube-api-access-47b97\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w\" (UID: \"c27ba098-46ba-4167-bedd-da6f0a16085e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w" Jul 07 20:12:09.693622 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:12:09.693577 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c27ba098-46ba-4167-bedd-da6f0a16085e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w\" (UID: \"c27ba098-46ba-4167-bedd-da6f0a16085e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w" Jul 07 20:12:09.693622 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:12:09.693599 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c27ba098-46ba-4167-bedd-da6f0a16085e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w\" (UID: \"c27ba098-46ba-4167-bedd-da6f0a16085e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w" Jul 07 20:12:09.693902 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:12:09.693882 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c27ba098-46ba-4167-bedd-da6f0a16085e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w\" (UID: \"c27ba098-46ba-4167-bedd-da6f0a16085e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w" Jul 07 20:12:09.693981 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:12:09.693958 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27ba098-46ba-4167-bedd-da6f0a16085e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w\" (UID: \"c27ba098-46ba-4167-bedd-da6f0a16085e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w" Jul 07 20:12:09.694031 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:12:09.694013 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c27ba098-46ba-4167-bedd-da6f0a16085e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w\" (UID: \"c27ba098-46ba-4167-bedd-da6f0a16085e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w" Jul 07 20:12:09.695670 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:12:09.695653 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c27ba098-46ba-4167-bedd-da6f0a16085e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w\" (UID: \"c27ba098-46ba-4167-bedd-da6f0a16085e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w" Jul 07 20:12:09.696031 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:12:09.696012 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c27ba098-46ba-4167-bedd-da6f0a16085e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w\" (UID: \"c27ba098-46ba-4167-bedd-da6f0a16085e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w" Jul 07 20:12:09.701517 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:12:09.701493 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47b97\" (UniqueName: \"kubernetes.io/projected/c27ba098-46ba-4167-bedd-da6f0a16085e-kube-api-access-47b97\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w\" (UID: \"c27ba098-46ba-4167-bedd-da6f0a16085e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w" Jul 07 20:12:09.870867 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:12:09.870771 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w" Jul 07 20:12:09.998366 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:12:09.998305 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w"] Jul 07 20:12:10.000716 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:12:10.000688 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27ba098_46ba_4167_bedd_da6f0a16085e.slice/crio-3b6b3c916f37144915803e4c61fedd835aca2e37c89bda06661604ee2d7cda24 WatchSource:0}: Error finding container 3b6b3c916f37144915803e4c61fedd835aca2e37c89bda06661604ee2d7cda24: Status 404 returned error can't find the container with id 3b6b3c916f37144915803e4c61fedd835aca2e37c89bda06661604ee2d7cda24 Jul 07 20:12:10.580198 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:12:10.580112 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w" event={"ID":"c27ba098-46ba-4167-bedd-da6f0a16085e","Type":"ContainerStarted","Data":"d8456d13f289d1ad8d5ca3b081c8f2e18bae9b3bcb18940a9f3f8d45c4e8f47b"} Jul 07 20:12:10.580198 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:12:10.580161 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w" event={"ID":"c27ba098-46ba-4167-bedd-da6f0a16085e","Type":"ContainerStarted","Data":"3b6b3c916f37144915803e4c61fedd835aca2e37c89bda06661604ee2d7cda24"} Jul 07 20:12:10.600832 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:12:10.600777 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w" podStartSLOduration=1.368097132 podStartE2EDuration="1.600760959s" totalImagesPullingTime="232.663827ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:12:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:12:10.5980777 +0000 UTC m=+650.752539654" watchObservedRunningTime="2026-07-07 20:12:10.600760959 +0000 UTC m=+650.755222908" Jul 07 20:12:19.871663 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:12:19.871630 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w" Jul 07 20:12:19.872118 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:12:19.871687 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w" Jul 07 20:12:19.884041 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:12:19.884013 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w" Jul 07 20:12:20.642341 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:12:20.642315 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w" Jul 07 20:15:00.143666 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:15:00.143578 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29724255-kzznf"] Jul 07 20:15:00.147048 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:15:00.147029 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724255-kzznf" Jul 07 20:15:00.154976 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:15:00.154949 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724255-kzznf"] Jul 07 20:15:00.235459 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:15:00.235422 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vkrk\" (UniqueName: \"kubernetes.io/projected/8ee5a14b-25d4-40c5-b111-7d5f89653d34-kube-api-access-6vkrk\") pod \"maas-api-key-cleanup-29724255-kzznf\" (UID: \"8ee5a14b-25d4-40c5-b111-7d5f89653d34\") " pod="opendatahub/maas-api-key-cleanup-29724255-kzznf" Jul 07 20:15:00.336536 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:15:00.336501 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vkrk\" (UniqueName: \"kubernetes.io/projected/8ee5a14b-25d4-40c5-b111-7d5f89653d34-kube-api-access-6vkrk\") pod \"maas-api-key-cleanup-29724255-kzznf\" (UID: \"8ee5a14b-25d4-40c5-b111-7d5f89653d34\") " pod="opendatahub/maas-api-key-cleanup-29724255-kzznf" Jul 07 20:15:00.346059 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:15:00.346036 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vkrk\" (UniqueName: \"kubernetes.io/projected/8ee5a14b-25d4-40c5-b111-7d5f89653d34-kube-api-access-6vkrk\") pod \"maas-api-key-cleanup-29724255-kzznf\" (UID: \"8ee5a14b-25d4-40c5-b111-7d5f89653d34\") " pod="opendatahub/maas-api-key-cleanup-29724255-kzznf" Jul 07 20:15:00.458695 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:15:00.458621 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724255-kzznf" Jul 07 20:15:00.588696 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:15:00.588673 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724255-kzznf"] Jul 07 20:15:00.590927 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:15:00.590899 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee5a14b_25d4_40c5_b111_7d5f89653d34.slice/crio-adf7865103db29ddd4b2b41335a168c0bab787bfe8a275ecb23dba6ae9ed94b0 WatchSource:0}: Error finding container adf7865103db29ddd4b2b41335a168c0bab787bfe8a275ecb23dba6ae9ed94b0: Status 404 returned error can't find the container with id adf7865103db29ddd4b2b41335a168c0bab787bfe8a275ecb23dba6ae9ed94b0 Jul 07 20:15:01.368373 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:15:01.368334 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724255-kzznf" event={"ID":"8ee5a14b-25d4-40c5-b111-7d5f89653d34","Type":"ContainerStarted","Data":"adf7865103db29ddd4b2b41335a168c0bab787bfe8a275ecb23dba6ae9ed94b0"} Jul 07 20:15:03.378667 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:15:03.378580 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724255-kzznf" event={"ID":"8ee5a14b-25d4-40c5-b111-7d5f89653d34","Type":"ContainerStarted","Data":"ac47c489bb7c8c725cc443291cb1a904827711d75b93d9c0c27fa6d031b6310c"} Jul 07 20:15:03.397035 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:15:03.396988 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29724255-kzznf" podStartSLOduration=1.276330012 podStartE2EDuration="3.396972281s" totalImagesPullingTime="2.120642269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:15:03.395453506 +0000 UTC m=+823.549915465" watchObservedRunningTime="2026-07-07 20:15:03.396972281 +0000 UTC m=+823.551434239" Jul 07 20:15:04.384813 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:15:04.384773 2565 generic.go:356] "Generic (PLEG): container finished" podID="8ee5a14b-25d4-40c5-b111-7d5f89653d34" containerID="ac47c489bb7c8c725cc443291cb1a904827711d75b93d9c0c27fa6d031b6310c" exitCode=0 Jul 07 20:15:04.385242 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:15:04.384853 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724255-kzznf" event={"ID":"8ee5a14b-25d4-40c5-b111-7d5f89653d34","Type":"ContainerDied","Data":"ac47c489bb7c8c725cc443291cb1a904827711d75b93d9c0c27fa6d031b6310c"} Jul 07 20:15:05.519699 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:15:05.519671 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724255-kzznf" Jul 07 20:15:05.595884 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:15:05.595853 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vkrk\" (UniqueName: \"kubernetes.io/projected/8ee5a14b-25d4-40c5-b111-7d5f89653d34-kube-api-access-6vkrk\") pod \"8ee5a14b-25d4-40c5-b111-7d5f89653d34\" (UID: \"8ee5a14b-25d4-40c5-b111-7d5f89653d34\") " Jul 07 20:15:05.597942 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:15:05.597910 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee5a14b-25d4-40c5-b111-7d5f89653d34-kube-api-access-6vkrk" (OuterVolumeSpecName: "kube-api-access-6vkrk") pod "8ee5a14b-25d4-40c5-b111-7d5f89653d34" (UID: "8ee5a14b-25d4-40c5-b111-7d5f89653d34"). InnerVolumeSpecName "kube-api-access-6vkrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 20:15:05.697466 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:15:05.697442 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vkrk\" (UniqueName: \"kubernetes.io/projected/8ee5a14b-25d4-40c5-b111-7d5f89653d34-kube-api-access-6vkrk\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:15:06.395824 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:15:06.395796 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724255-kzznf" Jul 07 20:15:06.395994 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:15:06.395828 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724255-kzznf" event={"ID":"8ee5a14b-25d4-40c5-b111-7d5f89653d34","Type":"ContainerDied","Data":"adf7865103db29ddd4b2b41335a168c0bab787bfe8a275ecb23dba6ae9ed94b0"} Jul 07 20:15:06.395994 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:15:06.395868 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adf7865103db29ddd4b2b41335a168c0bab787bfe8a275ecb23dba6ae9ed94b0" Jul 07 20:27:04.210071 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:04.210033 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q"] Jul 07 20:27:04.210496 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:04.210438 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ee5a14b-25d4-40c5-b111-7d5f89653d34" containerName="cleanup" Jul 07 20:27:04.210496 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:04.210450 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ee5a14b-25d4-40c5-b111-7d5f89653d34" containerName="cleanup" Jul 07 20:27:04.210568 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:04.210515 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ee5a14b-25d4-40c5-b111-7d5f89653d34" containerName="cleanup" Jul 07 20:27:04.213556 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:04.213539 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q" Jul 07 20:27:04.215767 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:04.215744 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-a0845fbd-serving-cert\"" Jul 07 20:27:04.222098 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:04.221854 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q"] Jul 07 20:27:04.296479 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:04.296436 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7wzz\" (UniqueName: \"kubernetes.io/projected/1245dd8b-4fbc-4665-82fd-05ed98122147-kube-api-access-r7wzz\") pod \"maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q\" (UID: \"1245dd8b-4fbc-4665-82fd-05ed98122147\") " pod="opendatahub/maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q" Jul 07 20:27:04.296695 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:04.296529 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1245dd8b-4fbc-4665-82fd-05ed98122147-openshift-service-ca\") pod \"maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q\" (UID: \"1245dd8b-4fbc-4665-82fd-05ed98122147\") " pod="opendatahub/maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q" Jul 07 20:27:04.296695 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:04.296557 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1245dd8b-4fbc-4665-82fd-05ed98122147-maas-api-tls\") pod \"maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q\" (UID: \"1245dd8b-4fbc-4665-82fd-05ed98122147\") " pod="opendatahub/maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q" Jul 07 20:27:04.398166 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:04.398119 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7wzz\" (UniqueName: \"kubernetes.io/projected/1245dd8b-4fbc-4665-82fd-05ed98122147-kube-api-access-r7wzz\") pod \"maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q\" (UID: \"1245dd8b-4fbc-4665-82fd-05ed98122147\") " pod="opendatahub/maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q" Jul 07 20:27:04.398356 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:04.398205 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1245dd8b-4fbc-4665-82fd-05ed98122147-openshift-service-ca\") pod \"maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q\" (UID: \"1245dd8b-4fbc-4665-82fd-05ed98122147\") " pod="opendatahub/maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q" Jul 07 20:27:04.398356 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:04.398238 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1245dd8b-4fbc-4665-82fd-05ed98122147-maas-api-tls\") pod \"maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q\" (UID: \"1245dd8b-4fbc-4665-82fd-05ed98122147\") " pod="opendatahub/maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q" Jul 07 20:27:04.398908 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:04.398884 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1245dd8b-4fbc-4665-82fd-05ed98122147-openshift-service-ca\") pod \"maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q\" (UID: \"1245dd8b-4fbc-4665-82fd-05ed98122147\") " pod="opendatahub/maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q" Jul 07 20:27:04.400714 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:04.400693 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1245dd8b-4fbc-4665-82fd-05ed98122147-maas-api-tls\") pod \"maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q\" (UID: \"1245dd8b-4fbc-4665-82fd-05ed98122147\") " pod="opendatahub/maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q" Jul 07 20:27:04.406663 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:04.406641 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7wzz\" (UniqueName: \"kubernetes.io/projected/1245dd8b-4fbc-4665-82fd-05ed98122147-kube-api-access-r7wzz\") pod \"maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q\" (UID: \"1245dd8b-4fbc-4665-82fd-05ed98122147\") " pod="opendatahub/maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q" Jul 07 20:27:04.526196 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:04.526105 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q" Jul 07 20:27:04.659700 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:04.659675 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q"] Jul 07 20:27:04.662527 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:27:04.662500 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1245dd8b_4fbc_4665_82fd_05ed98122147.slice/crio-b75e7f3f71a348385d51f90c001f94cba8da044732ab5fc760c887ababd03547 WatchSource:0}: Error finding container b75e7f3f71a348385d51f90c001f94cba8da044732ab5fc760c887ababd03547: Status 404 returned error can't find the container with id b75e7f3f71a348385d51f90c001f94cba8da044732ab5fc760c887ababd03547 Jul 07 20:27:04.664221 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:04.664205 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 20:27:04.853753 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:04.853667 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q" event={"ID":"1245dd8b-4fbc-4665-82fd-05ed98122147","Type":"ContainerStarted","Data":"b75e7f3f71a348385d51f90c001f94cba8da044732ab5fc760c887ababd03547"} Jul 07 20:27:05.859908 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:05.859867 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q" event={"ID":"1245dd8b-4fbc-4665-82fd-05ed98122147","Type":"ContainerStarted","Data":"d754379bcec7ad12d3a677cc6a1739187fa7d95e59cd1addbd23bf068421b800"} Jul 07 20:27:05.860289 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:05.859968 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q" Jul 07 20:27:05.878270 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:05.878219 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q" podStartSLOduration=1.2057870099999999 podStartE2EDuration="1.878201721s" totalImagesPullingTime="672.414711ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:27:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:27:05.876571001 +0000 UTC m=+1546.031032950" watchObservedRunningTime="2026-07-07 20:27:05.878201721 +0000 UTC m=+1546.032663670" Jul 07 20:27:09.616239 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:09.616201 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q"] Jul 07 20:27:09.616743 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:09.616431 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q" podUID="1245dd8b-4fbc-4665-82fd-05ed98122147" containerName="maas-api" containerID="cri-o://d754379bcec7ad12d3a677cc6a1739187fa7d95e59cd1addbd23bf068421b800" gracePeriod=30 Jul 07 20:27:09.870007 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:09.869933 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q" Jul 07 20:27:09.879770 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:09.879728 2565 generic.go:356] "Generic (PLEG): container finished" podID="1245dd8b-4fbc-4665-82fd-05ed98122147" containerID="d754379bcec7ad12d3a677cc6a1739187fa7d95e59cd1addbd23bf068421b800" exitCode=0 Jul 07 20:27:09.879770 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:09.879781 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q" Jul 07 20:27:09.879987 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:09.879794 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q" event={"ID":"1245dd8b-4fbc-4665-82fd-05ed98122147","Type":"ContainerDied","Data":"d754379bcec7ad12d3a677cc6a1739187fa7d95e59cd1addbd23bf068421b800"} Jul 07 20:27:09.879987 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:09.879849 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q" event={"ID":"1245dd8b-4fbc-4665-82fd-05ed98122147","Type":"ContainerDied","Data":"b75e7f3f71a348385d51f90c001f94cba8da044732ab5fc760c887ababd03547"} Jul 07 20:27:09.879987 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:09.879878 2565 scope.go:117] "RemoveContainer" containerID="d754379bcec7ad12d3a677cc6a1739187fa7d95e59cd1addbd23bf068421b800" Jul 07 20:27:09.889480 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:09.889452 2565 scope.go:117] "RemoveContainer" containerID="d754379bcec7ad12d3a677cc6a1739187fa7d95e59cd1addbd23bf068421b800" Jul 07 20:27:09.889855 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:27:09.889830 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d754379bcec7ad12d3a677cc6a1739187fa7d95e59cd1addbd23bf068421b800\": container with ID starting with d754379bcec7ad12d3a677cc6a1739187fa7d95e59cd1addbd23bf068421b800 not found: ID does not exist" containerID="d754379bcec7ad12d3a677cc6a1739187fa7d95e59cd1addbd23bf068421b800" Jul 07 20:27:09.889960 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:09.889867 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d754379bcec7ad12d3a677cc6a1739187fa7d95e59cd1addbd23bf068421b800"} err="failed to get container status \"d754379bcec7ad12d3a677cc6a1739187fa7d95e59cd1addbd23bf068421b800\": rpc error: code = NotFound desc = could not find container \"d754379bcec7ad12d3a677cc6a1739187fa7d95e59cd1addbd23bf068421b800\": container with ID starting with d754379bcec7ad12d3a677cc6a1739187fa7d95e59cd1addbd23bf068421b800 not found: ID does not exist" Jul 07 20:27:09.958659 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:09.958596 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1245dd8b-4fbc-4665-82fd-05ed98122147-openshift-service-ca\") pod \"1245dd8b-4fbc-4665-82fd-05ed98122147\" (UID: \"1245dd8b-4fbc-4665-82fd-05ed98122147\") " Jul 07 20:27:09.958861 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:09.958708 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7wzz\" (UniqueName: \"kubernetes.io/projected/1245dd8b-4fbc-4665-82fd-05ed98122147-kube-api-access-r7wzz\") pod \"1245dd8b-4fbc-4665-82fd-05ed98122147\" (UID: \"1245dd8b-4fbc-4665-82fd-05ed98122147\") " Jul 07 20:27:09.958861 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:09.958757 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1245dd8b-4fbc-4665-82fd-05ed98122147-maas-api-tls\") pod \"1245dd8b-4fbc-4665-82fd-05ed98122147\" (UID: \"1245dd8b-4fbc-4665-82fd-05ed98122147\") " Jul 07 20:27:09.959070 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:09.959042 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1245dd8b-4fbc-4665-82fd-05ed98122147-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1245dd8b-4fbc-4665-82fd-05ed98122147" (UID: "1245dd8b-4fbc-4665-82fd-05ed98122147"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 20:27:09.960935 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:09.960905 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1245dd8b-4fbc-4665-82fd-05ed98122147-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1245dd8b-4fbc-4665-82fd-05ed98122147" (UID: "1245dd8b-4fbc-4665-82fd-05ed98122147"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:27:09.961047 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:09.960909 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1245dd8b-4fbc-4665-82fd-05ed98122147-kube-api-access-r7wzz" (OuterVolumeSpecName: "kube-api-access-r7wzz") pod "1245dd8b-4fbc-4665-82fd-05ed98122147" (UID: "1245dd8b-4fbc-4665-82fd-05ed98122147"). InnerVolumeSpecName "kube-api-access-r7wzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 20:27:10.059662 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:10.059607 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1245dd8b-4fbc-4665-82fd-05ed98122147-maas-api-tls\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:27:10.059662 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:10.059654 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1245dd8b-4fbc-4665-82fd-05ed98122147-openshift-service-ca\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:27:10.059662 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:10.059664 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7wzz\" (UniqueName: \"kubernetes.io/projected/1245dd8b-4fbc-4665-82fd-05ed98122147-kube-api-access-r7wzz\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:27:10.203148 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:10.203112 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q"] Jul 07 20:27:10.208891 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:10.208858 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-a0845fbd-5c68bc6d86-kw56q"] Jul 07 20:27:11.685313 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:11.685276 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1245dd8b-4fbc-4665-82fd-05ed98122147" path="/var/lib/kubelet/pods/1245dd8b-4fbc-4665-82fd-05ed98122147/volumes" Jul 07 20:27:17.210088 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:17.210047 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-557e05f1-797c884878-g9twc"] Jul 07 20:27:17.211540 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:17.211514 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1245dd8b-4fbc-4665-82fd-05ed98122147" containerName="maas-api" Jul 07 20:27:17.211695 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:17.211684 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1245dd8b-4fbc-4665-82fd-05ed98122147" containerName="maas-api" Jul 07 20:27:17.212006 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:17.211991 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1245dd8b-4fbc-4665-82fd-05ed98122147" containerName="maas-api" Jul 07 20:27:17.220141 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:17.220114 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-557e05f1-797c884878-g9twc" Jul 07 20:27:17.223175 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:17.223147 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-557e05f1-serving-cert\"" Jul 07 20:27:17.223317 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:17.223215 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-557e05f1-797c884878-g9twc"] Jul 07 20:27:17.331354 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:17.331313 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpkwv\" (UniqueName: \"kubernetes.io/projected/0a16becd-d22f-414d-93df-f1060f71722b-kube-api-access-vpkwv\") pod \"maas-api-e2e-ait-557e05f1-797c884878-g9twc\" (UID: \"0a16becd-d22f-414d-93df-f1060f71722b\") " pod="opendatahub/maas-api-e2e-ait-557e05f1-797c884878-g9twc" Jul 07 20:27:17.331555 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:17.331366 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a16becd-d22f-414d-93df-f1060f71722b-maas-api-tls\") pod \"maas-api-e2e-ait-557e05f1-797c884878-g9twc\" (UID: \"0a16becd-d22f-414d-93df-f1060f71722b\") " pod="opendatahub/maas-api-e2e-ait-557e05f1-797c884878-g9twc" Jul 07 20:27:17.331555 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:17.331460 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a16becd-d22f-414d-93df-f1060f71722b-openshift-service-ca\") pod \"maas-api-e2e-ait-557e05f1-797c884878-g9twc\" (UID: \"0a16becd-d22f-414d-93df-f1060f71722b\") " pod="opendatahub/maas-api-e2e-ait-557e05f1-797c884878-g9twc" Jul 07 20:27:17.432414 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:17.432364 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpkwv\" (UniqueName: \"kubernetes.io/projected/0a16becd-d22f-414d-93df-f1060f71722b-kube-api-access-vpkwv\") pod \"maas-api-e2e-ait-557e05f1-797c884878-g9twc\" (UID: \"0a16becd-d22f-414d-93df-f1060f71722b\") " pod="opendatahub/maas-api-e2e-ait-557e05f1-797c884878-g9twc" Jul 07 20:27:17.432606 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:17.432448 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a16becd-d22f-414d-93df-f1060f71722b-maas-api-tls\") pod \"maas-api-e2e-ait-557e05f1-797c884878-g9twc\" (UID: \"0a16becd-d22f-414d-93df-f1060f71722b\") " pod="opendatahub/maas-api-e2e-ait-557e05f1-797c884878-g9twc" Jul 07 20:27:17.432606 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:17.432488 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a16becd-d22f-414d-93df-f1060f71722b-openshift-service-ca\") pod \"maas-api-e2e-ait-557e05f1-797c884878-g9twc\" (UID: \"0a16becd-d22f-414d-93df-f1060f71722b\") " pod="opendatahub/maas-api-e2e-ait-557e05f1-797c884878-g9twc" Jul 07 20:27:17.433300 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:17.433271 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a16becd-d22f-414d-93df-f1060f71722b-openshift-service-ca\") pod \"maas-api-e2e-ait-557e05f1-797c884878-g9twc\" (UID: \"0a16becd-d22f-414d-93df-f1060f71722b\") " pod="opendatahub/maas-api-e2e-ait-557e05f1-797c884878-g9twc" Jul 07 20:27:17.434991 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:17.434959 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a16becd-d22f-414d-93df-f1060f71722b-maas-api-tls\") pod \"maas-api-e2e-ait-557e05f1-797c884878-g9twc\" (UID: \"0a16becd-d22f-414d-93df-f1060f71722b\") " pod="opendatahub/maas-api-e2e-ait-557e05f1-797c884878-g9twc" Jul 07 20:27:17.449071 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:17.449039 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpkwv\" (UniqueName: \"kubernetes.io/projected/0a16becd-d22f-414d-93df-f1060f71722b-kube-api-access-vpkwv\") pod \"maas-api-e2e-ait-557e05f1-797c884878-g9twc\" (UID: \"0a16becd-d22f-414d-93df-f1060f71722b\") " pod="opendatahub/maas-api-e2e-ait-557e05f1-797c884878-g9twc" Jul 07 20:27:17.533225 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:17.533119 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-557e05f1-797c884878-g9twc" Jul 07 20:27:17.685487 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:27:17.685436 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a16becd_d22f_414d_93df_f1060f71722b.slice/crio-e92f850d8bce95b6ffec1421151beb730e155c4f394a3e96dd6cc8a6f23b38af WatchSource:0}: Error finding container e92f850d8bce95b6ffec1421151beb730e155c4f394a3e96dd6cc8a6f23b38af: Status 404 returned error can't find the container with id e92f850d8bce95b6ffec1421151beb730e155c4f394a3e96dd6cc8a6f23b38af Jul 07 20:27:17.686806 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:17.686779 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-557e05f1-797c884878-g9twc"] Jul 07 20:27:17.921506 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:17.921465 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-557e05f1-797c884878-g9twc" event={"ID":"0a16becd-d22f-414d-93df-f1060f71722b","Type":"ContainerStarted","Data":"e92f850d8bce95b6ffec1421151beb730e155c4f394a3e96dd6cc8a6f23b38af"} Jul 07 20:27:18.927499 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:18.927462 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-557e05f1-797c884878-g9twc" event={"ID":"0a16becd-d22f-414d-93df-f1060f71722b","Type":"ContainerStarted","Data":"ac55e9f92655560c35f9c7b25f20843dac3c4bdab20dcd28797a3db77bbfd899"} Jul 07 20:27:18.927911 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:18.927529 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-557e05f1-797c884878-g9twc" Jul 07 20:27:18.946888 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:18.946826 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-557e05f1-797c884878-g9twc" podStartSLOduration=1.529396735 podStartE2EDuration="1.946804067s" totalImagesPullingTime="417.407332ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:27:18.945963724 +0000 UTC m=+1559.100425675" watchObservedRunningTime="2026-07-07 20:27:18.946804067 +0000 UTC m=+1559.101266016" Jul 07 20:27:22.678261 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:22.678211 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-557e05f1-797c884878-g9twc"] Jul 07 20:27:22.679927 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:22.679893 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-557e05f1-797c884878-g9twc" podUID="0a16becd-d22f-414d-93df-f1060f71722b" containerName="maas-api" containerID="cri-o://ac55e9f92655560c35f9c7b25f20843dac3c4bdab20dcd28797a3db77bbfd899" gracePeriod=30 Jul 07 20:27:22.937213 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:22.937145 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-557e05f1-797c884878-g9twc" Jul 07 20:27:22.947882 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:22.947847 2565 generic.go:356] "Generic (PLEG): container finished" podID="0a16becd-d22f-414d-93df-f1060f71722b" containerID="ac55e9f92655560c35f9c7b25f20843dac3c4bdab20dcd28797a3db77bbfd899" exitCode=0 Jul 07 20:27:22.948045 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:22.947891 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-557e05f1-797c884878-g9twc" Jul 07 20:27:22.948045 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:22.947919 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-557e05f1-797c884878-g9twc" event={"ID":"0a16becd-d22f-414d-93df-f1060f71722b","Type":"ContainerDied","Data":"ac55e9f92655560c35f9c7b25f20843dac3c4bdab20dcd28797a3db77bbfd899"} Jul 07 20:27:22.948045 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:22.947969 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-557e05f1-797c884878-g9twc" event={"ID":"0a16becd-d22f-414d-93df-f1060f71722b","Type":"ContainerDied","Data":"e92f850d8bce95b6ffec1421151beb730e155c4f394a3e96dd6cc8a6f23b38af"} Jul 07 20:27:22.948553 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:22.948203 2565 scope.go:117] "RemoveContainer" containerID="ac55e9f92655560c35f9c7b25f20843dac3c4bdab20dcd28797a3db77bbfd899" Jul 07 20:27:22.965127 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:22.965106 2565 scope.go:117] "RemoveContainer" containerID="ac55e9f92655560c35f9c7b25f20843dac3c4bdab20dcd28797a3db77bbfd899" Jul 07 20:27:22.965487 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:27:22.965460 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac55e9f92655560c35f9c7b25f20843dac3c4bdab20dcd28797a3db77bbfd899\": container with ID starting with ac55e9f92655560c35f9c7b25f20843dac3c4bdab20dcd28797a3db77bbfd899 not found: ID does not exist" containerID="ac55e9f92655560c35f9c7b25f20843dac3c4bdab20dcd28797a3db77bbfd899" Jul 07 20:27:22.965577 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:22.965499 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac55e9f92655560c35f9c7b25f20843dac3c4bdab20dcd28797a3db77bbfd899"} err="failed to get container status \"ac55e9f92655560c35f9c7b25f20843dac3c4bdab20dcd28797a3db77bbfd899\": rpc error: code = NotFound desc = could not find container \"ac55e9f92655560c35f9c7b25f20843dac3c4bdab20dcd28797a3db77bbfd899\": container with ID starting with ac55e9f92655560c35f9c7b25f20843dac3c4bdab20dcd28797a3db77bbfd899 not found: ID does not exist" Jul 07 20:27:22.984965 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:22.984929 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a16becd-d22f-414d-93df-f1060f71722b-openshift-service-ca\") pod \"0a16becd-d22f-414d-93df-f1060f71722b\" (UID: \"0a16becd-d22f-414d-93df-f1060f71722b\") " Jul 07 20:27:22.985119 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:22.985021 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpkwv\" (UniqueName: \"kubernetes.io/projected/0a16becd-d22f-414d-93df-f1060f71722b-kube-api-access-vpkwv\") pod \"0a16becd-d22f-414d-93df-f1060f71722b\" (UID: \"0a16becd-d22f-414d-93df-f1060f71722b\") " Jul 07 20:27:22.985119 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:22.985058 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a16becd-d22f-414d-93df-f1060f71722b-maas-api-tls\") pod \"0a16becd-d22f-414d-93df-f1060f71722b\" (UID: \"0a16becd-d22f-414d-93df-f1060f71722b\") " Jul 07 20:27:22.985305 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:22.985279 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a16becd-d22f-414d-93df-f1060f71722b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0a16becd-d22f-414d-93df-f1060f71722b" (UID: "0a16becd-d22f-414d-93df-f1060f71722b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 20:27:22.987179 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:22.987149 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a16becd-d22f-414d-93df-f1060f71722b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0a16becd-d22f-414d-93df-f1060f71722b" (UID: "0a16becd-d22f-414d-93df-f1060f71722b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:27:22.987284 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:22.987193 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a16becd-d22f-414d-93df-f1060f71722b-kube-api-access-vpkwv" (OuterVolumeSpecName: "kube-api-access-vpkwv") pod "0a16becd-d22f-414d-93df-f1060f71722b" (UID: "0a16becd-d22f-414d-93df-f1060f71722b"). InnerVolumeSpecName "kube-api-access-vpkwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 20:27:23.086269 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:23.086223 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpkwv\" (UniqueName: \"kubernetes.io/projected/0a16becd-d22f-414d-93df-f1060f71722b-kube-api-access-vpkwv\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:27:23.086269 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:23.086263 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a16becd-d22f-414d-93df-f1060f71722b-maas-api-tls\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:27:23.086269 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:23.086274 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a16becd-d22f-414d-93df-f1060f71722b-openshift-service-ca\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:27:23.272666 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:23.272627 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-557e05f1-797c884878-g9twc"] Jul 07 20:27:23.278760 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:23.278726 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-557e05f1-797c884878-g9twc"] Jul 07 20:27:23.685165 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:23.685125 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a16becd-d22f-414d-93df-f1060f71722b" path="/var/lib/kubelet/pods/0a16becd-d22f-414d-93df-f1060f71722b/volumes" Jul 07 20:27:35.993785 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:35.993749 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj"] Jul 07 20:27:35.994361 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:35.994338 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a16becd-d22f-414d-93df-f1060f71722b" containerName="maas-api" Jul 07 20:27:35.994471 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:35.994365 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a16becd-d22f-414d-93df-f1060f71722b" containerName="maas-api" Jul 07 20:27:35.994529 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:35.994493 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a16becd-d22f-414d-93df-f1060f71722b" containerName="maas-api" Jul 07 20:27:35.999068 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:35.999032 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj" Jul 07 20:27:36.001372 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:36.001347 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-70f9ecd0-serving-cert\"" Jul 07 20:27:36.006577 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:36.006543 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj"] Jul 07 20:27:36.108778 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:36.108733 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/414dffa0-c49c-4717-9c44-9d69ef371a70-openshift-service-ca\") pod \"maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj\" (UID: \"414dffa0-c49c-4717-9c44-9d69ef371a70\") " pod="opendatahub/maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj" Jul 07 20:27:36.108993 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:36.108794 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/414dffa0-c49c-4717-9c44-9d69ef371a70-maas-api-tls\") pod \"maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj\" (UID: \"414dffa0-c49c-4717-9c44-9d69ef371a70\") " pod="opendatahub/maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj" Jul 07 20:27:36.108993 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:36.108922 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vfd4\" (UniqueName: \"kubernetes.io/projected/414dffa0-c49c-4717-9c44-9d69ef371a70-kube-api-access-5vfd4\") pod \"maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj\" (UID: \"414dffa0-c49c-4717-9c44-9d69ef371a70\") " pod="opendatahub/maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj" Jul 07 20:27:36.209624 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:36.209586 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vfd4\" (UniqueName: \"kubernetes.io/projected/414dffa0-c49c-4717-9c44-9d69ef371a70-kube-api-access-5vfd4\") pod \"maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj\" (UID: \"414dffa0-c49c-4717-9c44-9d69ef371a70\") " pod="opendatahub/maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj" Jul 07 20:27:36.209831 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:36.209651 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/414dffa0-c49c-4717-9c44-9d69ef371a70-openshift-service-ca\") pod \"maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj\" (UID: \"414dffa0-c49c-4717-9c44-9d69ef371a70\") " pod="opendatahub/maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj" Jul 07 20:27:36.209831 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:36.209697 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/414dffa0-c49c-4717-9c44-9d69ef371a70-maas-api-tls\") pod \"maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj\" (UID: \"414dffa0-c49c-4717-9c44-9d69ef371a70\") " pod="opendatahub/maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj" Jul 07 20:27:36.210275 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:36.210251 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/414dffa0-c49c-4717-9c44-9d69ef371a70-openshift-service-ca\") pod \"maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj\" (UID: \"414dffa0-c49c-4717-9c44-9d69ef371a70\") " pod="opendatahub/maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj" Jul 07 20:27:36.212033 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:36.212015 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/414dffa0-c49c-4717-9c44-9d69ef371a70-maas-api-tls\") pod \"maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj\" (UID: \"414dffa0-c49c-4717-9c44-9d69ef371a70\") " pod="opendatahub/maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj" Jul 07 20:27:36.217801 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:36.217780 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vfd4\" (UniqueName: \"kubernetes.io/projected/414dffa0-c49c-4717-9c44-9d69ef371a70-kube-api-access-5vfd4\") pod \"maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj\" (UID: \"414dffa0-c49c-4717-9c44-9d69ef371a70\") " pod="opendatahub/maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj" Jul 07 20:27:36.312938 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:36.312837 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj" Jul 07 20:27:36.456052 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:36.456026 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj"] Jul 07 20:27:36.458765 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:27:36.458717 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod414dffa0_c49c_4717_9c44_9d69ef371a70.slice/crio-82bad27babdbe198eb792a26419a23f30f0d4de72a40f6ad0745c8fadeeb10c2 WatchSource:0}: Error finding container 82bad27babdbe198eb792a26419a23f30f0d4de72a40f6ad0745c8fadeeb10c2: Status 404 returned error can't find the container with id 82bad27babdbe198eb792a26419a23f30f0d4de72a40f6ad0745c8fadeeb10c2 Jul 07 20:27:37.018971 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:37.018931 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj" event={"ID":"414dffa0-c49c-4717-9c44-9d69ef371a70","Type":"ContainerStarted","Data":"344325bb09ee9cc436278b984ce0c41e6fd2c63ae4694009559da334d9da0ee1"} Jul 07 20:27:37.019493 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:37.018994 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj" Jul 07 20:27:37.019493 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:37.019014 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj" event={"ID":"414dffa0-c49c-4717-9c44-9d69ef371a70","Type":"ContainerStarted","Data":"82bad27babdbe198eb792a26419a23f30f0d4de72a40f6ad0745c8fadeeb10c2"} Jul 07 20:27:37.038303 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:37.038245 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj" podStartSLOduration=1.646223726 podStartE2EDuration="2.038227193s" totalImagesPullingTime="392.003467ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:27:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:27:37.036124771 +0000 UTC m=+1577.190586721" watchObservedRunningTime="2026-07-07 20:27:37.038227193 +0000 UTC m=+1577.192689141" Jul 07 20:27:40.780302 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:40.780264 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj"] Jul 07 20:27:40.780851 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:40.780526 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj" podUID="414dffa0-c49c-4717-9c44-9d69ef371a70" containerName="maas-api" containerID="cri-o://344325bb09ee9cc436278b984ce0c41e6fd2c63ae4694009559da334d9da0ee1" gracePeriod=30 Jul 07 20:27:41.033154 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:41.033086 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj" Jul 07 20:27:41.040589 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:41.040554 2565 generic.go:356] "Generic (PLEG): container finished" podID="414dffa0-c49c-4717-9c44-9d69ef371a70" containerID="344325bb09ee9cc436278b984ce0c41e6fd2c63ae4694009559da334d9da0ee1" exitCode=0 Jul 07 20:27:41.040756 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:41.040597 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj" Jul 07 20:27:41.040756 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:41.040651 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj" event={"ID":"414dffa0-c49c-4717-9c44-9d69ef371a70","Type":"ContainerDied","Data":"344325bb09ee9cc436278b984ce0c41e6fd2c63ae4694009559da334d9da0ee1"} Jul 07 20:27:41.040756 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:41.040710 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj" event={"ID":"414dffa0-c49c-4717-9c44-9d69ef371a70","Type":"ContainerDied","Data":"82bad27babdbe198eb792a26419a23f30f0d4de72a40f6ad0745c8fadeeb10c2"} Jul 07 20:27:41.040756 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:41.040736 2565 scope.go:117] "RemoveContainer" containerID="344325bb09ee9cc436278b984ce0c41e6fd2c63ae4694009559da334d9da0ee1" Jul 07 20:27:41.049691 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:41.049665 2565 scope.go:117] "RemoveContainer" containerID="344325bb09ee9cc436278b984ce0c41e6fd2c63ae4694009559da334d9da0ee1" Jul 07 20:27:41.050037 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:27:41.050008 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"344325bb09ee9cc436278b984ce0c41e6fd2c63ae4694009559da334d9da0ee1\": container with ID starting with 344325bb09ee9cc436278b984ce0c41e6fd2c63ae4694009559da334d9da0ee1 not found: ID does not exist" containerID="344325bb09ee9cc436278b984ce0c41e6fd2c63ae4694009559da334d9da0ee1" Jul 07 20:27:41.050119 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:41.050049 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"344325bb09ee9cc436278b984ce0c41e6fd2c63ae4694009559da334d9da0ee1"} err="failed to get container status \"344325bb09ee9cc436278b984ce0c41e6fd2c63ae4694009559da334d9da0ee1\": rpc error: code = NotFound desc = could not find container \"344325bb09ee9cc436278b984ce0c41e6fd2c63ae4694009559da334d9da0ee1\": container with ID starting with 344325bb09ee9cc436278b984ce0c41e6fd2c63ae4694009559da334d9da0ee1 not found: ID does not exist" Jul 07 20:27:41.165005 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:41.164965 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/414dffa0-c49c-4717-9c44-9d69ef371a70-maas-api-tls\") pod \"414dffa0-c49c-4717-9c44-9d69ef371a70\" (UID: \"414dffa0-c49c-4717-9c44-9d69ef371a70\") " Jul 07 20:27:41.165197 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:41.165027 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vfd4\" (UniqueName: \"kubernetes.io/projected/414dffa0-c49c-4717-9c44-9d69ef371a70-kube-api-access-5vfd4\") pod \"414dffa0-c49c-4717-9c44-9d69ef371a70\" (UID: \"414dffa0-c49c-4717-9c44-9d69ef371a70\") " Jul 07 20:27:41.165197 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:41.165068 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/414dffa0-c49c-4717-9c44-9d69ef371a70-openshift-service-ca\") pod \"414dffa0-c49c-4717-9c44-9d69ef371a70\" (UID: \"414dffa0-c49c-4717-9c44-9d69ef371a70\") " Jul 07 20:27:41.165465 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:41.165430 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/414dffa0-c49c-4717-9c44-9d69ef371a70-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "414dffa0-c49c-4717-9c44-9d69ef371a70" (UID: "414dffa0-c49c-4717-9c44-9d69ef371a70"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 20:27:41.167225 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:41.167198 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/414dffa0-c49c-4717-9c44-9d69ef371a70-kube-api-access-5vfd4" (OuterVolumeSpecName: "kube-api-access-5vfd4") pod "414dffa0-c49c-4717-9c44-9d69ef371a70" (UID: "414dffa0-c49c-4717-9c44-9d69ef371a70"). InnerVolumeSpecName "kube-api-access-5vfd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 20:27:41.167303 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:41.167220 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/414dffa0-c49c-4717-9c44-9d69ef371a70-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "414dffa0-c49c-4717-9c44-9d69ef371a70" (UID: "414dffa0-c49c-4717-9c44-9d69ef371a70"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:27:41.266811 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:41.266765 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/414dffa0-c49c-4717-9c44-9d69ef371a70-openshift-service-ca\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:27:41.266811 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:41.266802 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/414dffa0-c49c-4717-9c44-9d69ef371a70-maas-api-tls\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:27:41.266811 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:41.266811 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vfd4\" (UniqueName: \"kubernetes.io/projected/414dffa0-c49c-4717-9c44-9d69ef371a70-kube-api-access-5vfd4\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:27:41.363377 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:41.363337 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj"] Jul 07 20:27:41.369306 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:41.369273 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-70f9ecd0-5b8c5cb987-qpprj"] Jul 07 20:27:41.685111 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:27:41.685068 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="414dffa0-c49c-4717-9c44-9d69ef371a70" path="/var/lib/kubelet/pods/414dffa0-c49c-4717-9c44-9d69ef371a70/volumes" Jul 07 20:29:28.300099 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:28.300061 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-e8652739-57b69cbc86-bf797"] Jul 07 20:29:28.302726 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:28.300541 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="414dffa0-c49c-4717-9c44-9d69ef371a70" containerName="maas-api" Jul 07 20:29:28.302726 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:28.300561 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="414dffa0-c49c-4717-9c44-9d69ef371a70" containerName="maas-api" Jul 07 20:29:28.302726 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:28.300698 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="414dffa0-c49c-4717-9c44-9d69ef371a70" containerName="maas-api" Jul 07 20:29:28.303799 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:28.303781 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-e8652739-57b69cbc86-bf797" Jul 07 20:29:28.306442 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:28.306391 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-e8652739-serving-cert\"" Jul 07 20:29:28.314654 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:28.314630 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-e8652739-57b69cbc86-bf797"] Jul 07 20:29:28.357367 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:28.357326 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c287d2-322a-43a7-a0dc-a71bf4798a85-maas-api-tls\") pod \"maas-api-e2e-mt-e8652739-57b69cbc86-bf797\" (UID: \"a2c287d2-322a-43a7-a0dc-a71bf4798a85\") " pod="opendatahub/maas-api-e2e-mt-e8652739-57b69cbc86-bf797" Jul 07 20:29:28.357582 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:28.357377 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b257l\" (UniqueName: \"kubernetes.io/projected/a2c287d2-322a-43a7-a0dc-a71bf4798a85-kube-api-access-b257l\") pod \"maas-api-e2e-mt-e8652739-57b69cbc86-bf797\" (UID: \"a2c287d2-322a-43a7-a0dc-a71bf4798a85\") " pod="opendatahub/maas-api-e2e-mt-e8652739-57b69cbc86-bf797" Jul 07 20:29:28.357582 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:28.357527 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c287d2-322a-43a7-a0dc-a71bf4798a85-openshift-service-ca\") pod \"maas-api-e2e-mt-e8652739-57b69cbc86-bf797\" (UID: \"a2c287d2-322a-43a7-a0dc-a71bf4798a85\") " pod="opendatahub/maas-api-e2e-mt-e8652739-57b69cbc86-bf797" Jul 07 20:29:28.458033 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:28.457988 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c287d2-322a-43a7-a0dc-a71bf4798a85-openshift-service-ca\") pod \"maas-api-e2e-mt-e8652739-57b69cbc86-bf797\" (UID: \"a2c287d2-322a-43a7-a0dc-a71bf4798a85\") " pod="opendatahub/maas-api-e2e-mt-e8652739-57b69cbc86-bf797" Jul 07 20:29:28.458192 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:28.458044 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c287d2-322a-43a7-a0dc-a71bf4798a85-maas-api-tls\") pod \"maas-api-e2e-mt-e8652739-57b69cbc86-bf797\" (UID: \"a2c287d2-322a-43a7-a0dc-a71bf4798a85\") " pod="opendatahub/maas-api-e2e-mt-e8652739-57b69cbc86-bf797" Jul 07 20:29:28.458192 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:28.458077 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b257l\" (UniqueName: \"kubernetes.io/projected/a2c287d2-322a-43a7-a0dc-a71bf4798a85-kube-api-access-b257l\") pod \"maas-api-e2e-mt-e8652739-57b69cbc86-bf797\" (UID: \"a2c287d2-322a-43a7-a0dc-a71bf4798a85\") " pod="opendatahub/maas-api-e2e-mt-e8652739-57b69cbc86-bf797" Jul 07 20:29:28.458261 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:29:28.458208 2565 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-e8652739-serving-cert: secret "maas-api-e2e-mt-e8652739-serving-cert" not found Jul 07 20:29:28.458300 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:29:28.458290 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2c287d2-322a-43a7-a0dc-a71bf4798a85-maas-api-tls podName:a2c287d2-322a-43a7-a0dc-a71bf4798a85 nodeName:}" failed. No retries permitted until 2026-07-07 20:29:28.958266802 +0000 UTC m=+1689.112728750 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a2c287d2-322a-43a7-a0dc-a71bf4798a85-maas-api-tls") pod "maas-api-e2e-mt-e8652739-57b69cbc86-bf797" (UID: "a2c287d2-322a-43a7-a0dc-a71bf4798a85") : secret "maas-api-e2e-mt-e8652739-serving-cert" not found Jul 07 20:29:28.458788 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:28.458769 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c287d2-322a-43a7-a0dc-a71bf4798a85-openshift-service-ca\") pod \"maas-api-e2e-mt-e8652739-57b69cbc86-bf797\" (UID: \"a2c287d2-322a-43a7-a0dc-a71bf4798a85\") " pod="opendatahub/maas-api-e2e-mt-e8652739-57b69cbc86-bf797" Jul 07 20:29:28.468011 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:28.467983 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b257l\" (UniqueName: \"kubernetes.io/projected/a2c287d2-322a-43a7-a0dc-a71bf4798a85-kube-api-access-b257l\") pod \"maas-api-e2e-mt-e8652739-57b69cbc86-bf797\" (UID: \"a2c287d2-322a-43a7-a0dc-a71bf4798a85\") " pod="opendatahub/maas-api-e2e-mt-e8652739-57b69cbc86-bf797" Jul 07 20:29:28.962582 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:28.962541 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c287d2-322a-43a7-a0dc-a71bf4798a85-maas-api-tls\") pod \"maas-api-e2e-mt-e8652739-57b69cbc86-bf797\" (UID: \"a2c287d2-322a-43a7-a0dc-a71bf4798a85\") " pod="opendatahub/maas-api-e2e-mt-e8652739-57b69cbc86-bf797" Jul 07 20:29:28.965209 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:28.965178 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c287d2-322a-43a7-a0dc-a71bf4798a85-maas-api-tls\") pod \"maas-api-e2e-mt-e8652739-57b69cbc86-bf797\" (UID: \"a2c287d2-322a-43a7-a0dc-a71bf4798a85\") " pod="opendatahub/maas-api-e2e-mt-e8652739-57b69cbc86-bf797" Jul 07 20:29:29.215913 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:29.215812 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-e8652739-57b69cbc86-bf797" Jul 07 20:29:29.358293 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:29.358265 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-e8652739-57b69cbc86-bf797"] Jul 07 20:29:29.360183 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:29:29.360149 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c287d2_322a_43a7_a0dc_a71bf4798a85.slice/crio-429866571c1510589e55e85124b5cb8637fabcea1ebd46a127c2b5199a81f98d WatchSource:0}: Error finding container 429866571c1510589e55e85124b5cb8637fabcea1ebd46a127c2b5199a81f98d: Status 404 returned error can't find the container with id 429866571c1510589e55e85124b5cb8637fabcea1ebd46a127c2b5199a81f98d Jul 07 20:29:29.559522 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:29.559429 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-e8652739-57b69cbc86-bf797" event={"ID":"a2c287d2-322a-43a7-a0dc-a71bf4798a85","Type":"ContainerStarted","Data":"429866571c1510589e55e85124b5cb8637fabcea1ebd46a127c2b5199a81f98d"} Jul 07 20:29:30.566092 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:30.566056 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-e8652739-57b69cbc86-bf797" event={"ID":"a2c287d2-322a-43a7-a0dc-a71bf4798a85","Type":"ContainerStarted","Data":"e580d52d6757c52f45c0c69399ce6e2ffca8fffb71cc025f598193f020e7c5fb"} Jul 07 20:29:30.566499 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:30.566140 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-e8652739-57b69cbc86-bf797" Jul 07 20:29:30.585181 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:30.585121 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-e8652739-57b69cbc86-bf797" podStartSLOduration=2.016869051 podStartE2EDuration="2.585102574s" totalImagesPullingTime="568.233523ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:29:30.583922175 +0000 UTC m=+1690.738384126" watchObservedRunningTime="2026-07-07 20:29:30.585102574 +0000 UTC m=+1690.739564522" Jul 07 20:29:36.576480 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:36.576447 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-e8652739-57b69cbc86-bf797" Jul 07 20:29:41.155485 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:41.155451 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-e8652739-57b69cbc86-bf797"] Jul 07 20:29:41.155902 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:41.155657 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-e8652739-57b69cbc86-bf797" podUID="a2c287d2-322a-43a7-a0dc-a71bf4798a85" containerName="maas-api" containerID="cri-o://e580d52d6757c52f45c0c69399ce6e2ffca8fffb71cc025f598193f020e7c5fb" gracePeriod=30 Jul 07 20:29:41.407079 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:41.407016 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-e8652739-57b69cbc86-bf797" Jul 07 20:29:41.471653 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:41.471620 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b257l\" (UniqueName: \"kubernetes.io/projected/a2c287d2-322a-43a7-a0dc-a71bf4798a85-kube-api-access-b257l\") pod \"a2c287d2-322a-43a7-a0dc-a71bf4798a85\" (UID: \"a2c287d2-322a-43a7-a0dc-a71bf4798a85\") " Jul 07 20:29:41.471825 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:41.471713 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c287d2-322a-43a7-a0dc-a71bf4798a85-openshift-service-ca\") pod \"a2c287d2-322a-43a7-a0dc-a71bf4798a85\" (UID: \"a2c287d2-322a-43a7-a0dc-a71bf4798a85\") " Jul 07 20:29:41.471825 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:41.471741 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c287d2-322a-43a7-a0dc-a71bf4798a85-maas-api-tls\") pod \"a2c287d2-322a-43a7-a0dc-a71bf4798a85\" (UID: \"a2c287d2-322a-43a7-a0dc-a71bf4798a85\") " Jul 07 20:29:41.472132 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:41.472101 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c287d2-322a-43a7-a0dc-a71bf4798a85-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a2c287d2-322a-43a7-a0dc-a71bf4798a85" (UID: "a2c287d2-322a-43a7-a0dc-a71bf4798a85"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 20:29:41.473919 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:41.473889 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2c287d2-322a-43a7-a0dc-a71bf4798a85-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a2c287d2-322a-43a7-a0dc-a71bf4798a85" (UID: "a2c287d2-322a-43a7-a0dc-a71bf4798a85"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:29:41.474021 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:41.473911 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c287d2-322a-43a7-a0dc-a71bf4798a85-kube-api-access-b257l" (OuterVolumeSpecName: "kube-api-access-b257l") pod "a2c287d2-322a-43a7-a0dc-a71bf4798a85" (UID: "a2c287d2-322a-43a7-a0dc-a71bf4798a85"). InnerVolumeSpecName "kube-api-access-b257l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 20:29:41.573031 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:41.572991 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c287d2-322a-43a7-a0dc-a71bf4798a85-openshift-service-ca\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:29:41.573031 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:41.573026 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c287d2-322a-43a7-a0dc-a71bf4798a85-maas-api-tls\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:29:41.573031 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:41.573035 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b257l\" (UniqueName: \"kubernetes.io/projected/a2c287d2-322a-43a7-a0dc-a71bf4798a85-kube-api-access-b257l\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:29:41.621025 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:41.620988 2565 generic.go:356] "Generic (PLEG): container finished" podID="a2c287d2-322a-43a7-a0dc-a71bf4798a85" containerID="e580d52d6757c52f45c0c69399ce6e2ffca8fffb71cc025f598193f020e7c5fb" exitCode=0 Jul 07 20:29:41.621214 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:41.621043 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-e8652739-57b69cbc86-bf797" event={"ID":"a2c287d2-322a-43a7-a0dc-a71bf4798a85","Type":"ContainerDied","Data":"e580d52d6757c52f45c0c69399ce6e2ffca8fffb71cc025f598193f020e7c5fb"} Jul 07 20:29:41.621214 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:41.621048 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-e8652739-57b69cbc86-bf797" Jul 07 20:29:41.621214 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:41.621084 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-e8652739-57b69cbc86-bf797" event={"ID":"a2c287d2-322a-43a7-a0dc-a71bf4798a85","Type":"ContainerDied","Data":"429866571c1510589e55e85124b5cb8637fabcea1ebd46a127c2b5199a81f98d"} Jul 07 20:29:41.621214 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:41.621106 2565 scope.go:117] "RemoveContainer" containerID="e580d52d6757c52f45c0c69399ce6e2ffca8fffb71cc025f598193f020e7c5fb" Jul 07 20:29:41.630428 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:41.630383 2565 scope.go:117] "RemoveContainer" containerID="e580d52d6757c52f45c0c69399ce6e2ffca8fffb71cc025f598193f020e7c5fb" Jul 07 20:29:41.630719 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:29:41.630698 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e580d52d6757c52f45c0c69399ce6e2ffca8fffb71cc025f598193f020e7c5fb\": container with ID starting with e580d52d6757c52f45c0c69399ce6e2ffca8fffb71cc025f598193f020e7c5fb not found: ID does not exist" containerID="e580d52d6757c52f45c0c69399ce6e2ffca8fffb71cc025f598193f020e7c5fb" Jul 07 20:29:41.630807 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:41.630726 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e580d52d6757c52f45c0c69399ce6e2ffca8fffb71cc025f598193f020e7c5fb"} err="failed to get container status \"e580d52d6757c52f45c0c69399ce6e2ffca8fffb71cc025f598193f020e7c5fb\": rpc error: code = NotFound desc = could not find container \"e580d52d6757c52f45c0c69399ce6e2ffca8fffb71cc025f598193f020e7c5fb\": container with ID starting with e580d52d6757c52f45c0c69399ce6e2ffca8fffb71cc025f598193f020e7c5fb not found: ID does not exist" Jul 07 20:29:41.647192 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:41.647126 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-e8652739-57b69cbc86-bf797"] Jul 07 20:29:41.651519 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:41.651488 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-e8652739-57b69cbc86-bf797"] Jul 07 20:29:41.684734 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:29:41.684648 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2c287d2-322a-43a7-a0dc-a71bf4798a85" path="/var/lib/kubelet/pods/a2c287d2-322a-43a7-a0dc-a71bf4798a85/volumes" Jul 07 20:30:00.145341 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:00.145308 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29724270-qh4mr"] Jul 07 20:30:00.145827 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:00.145716 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2c287d2-322a-43a7-a0dc-a71bf4798a85" containerName="maas-api" Jul 07 20:30:00.145827 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:00.145729 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2c287d2-322a-43a7-a0dc-a71bf4798a85" containerName="maas-api" Jul 07 20:30:00.145827 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:00.145798 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2c287d2-322a-43a7-a0dc-a71bf4798a85" containerName="maas-api" Jul 07 20:30:00.148685 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:00.148668 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724270-qh4mr" Jul 07 20:30:00.158463 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:00.158428 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724270-qh4mr"] Jul 07 20:30:00.256120 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:00.256073 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn5c7\" (UniqueName: \"kubernetes.io/projected/0e431ffc-713b-41c8-bd1a-377110182a1c-kube-api-access-xn5c7\") pod \"maas-api-key-cleanup-29724270-qh4mr\" (UID: \"0e431ffc-713b-41c8-bd1a-377110182a1c\") " pod="opendatahub/maas-api-key-cleanup-29724270-qh4mr" Jul 07 20:30:00.357463 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:00.357382 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn5c7\" (UniqueName: \"kubernetes.io/projected/0e431ffc-713b-41c8-bd1a-377110182a1c-kube-api-access-xn5c7\") pod \"maas-api-key-cleanup-29724270-qh4mr\" (UID: \"0e431ffc-713b-41c8-bd1a-377110182a1c\") " pod="opendatahub/maas-api-key-cleanup-29724270-qh4mr" Jul 07 20:30:00.366620 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:00.366592 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn5c7\" (UniqueName: \"kubernetes.io/projected/0e431ffc-713b-41c8-bd1a-377110182a1c-kube-api-access-xn5c7\") pod \"maas-api-key-cleanup-29724270-qh4mr\" (UID: \"0e431ffc-713b-41c8-bd1a-377110182a1c\") " pod="opendatahub/maas-api-key-cleanup-29724270-qh4mr" Jul 07 20:30:00.460564 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:00.460467 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724270-qh4mr" Jul 07 20:30:00.595052 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:00.595025 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724270-qh4mr"] Jul 07 20:30:00.597431 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:30:00.597387 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e431ffc_713b_41c8_bd1a_377110182a1c.slice/crio-5da01e23335adf2640fc147992a736c50aef5719b306a6e968170e837b21a8d2 WatchSource:0}: Error finding container 5da01e23335adf2640fc147992a736c50aef5719b306a6e968170e837b21a8d2: Status 404 returned error can't find the container with id 5da01e23335adf2640fc147992a736c50aef5719b306a6e968170e837b21a8d2 Jul 07 20:30:00.720473 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:00.720367 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724270-qh4mr" event={"ID":"0e431ffc-713b-41c8-bd1a-377110182a1c","Type":"ContainerStarted","Data":"5da01e23335adf2640fc147992a736c50aef5719b306a6e968170e837b21a8d2"} Jul 07 20:30:01.726307 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:01.726267 2565 generic.go:356] "Generic (PLEG): container finished" podID="0e431ffc-713b-41c8-bd1a-377110182a1c" containerID="b35a3b8259ad26b7c7d5d20521a5d0174a323e70b4430198bf0c27336c40d875" exitCode=0 Jul 07 20:30:01.726738 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:01.726348 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724270-qh4mr" event={"ID":"0e431ffc-713b-41c8-bd1a-377110182a1c","Type":"ContainerDied","Data":"b35a3b8259ad26b7c7d5d20521a5d0174a323e70b4430198bf0c27336c40d875"} Jul 07 20:30:02.864236 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:02.864208 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724270-qh4mr" Jul 07 20:30:02.984569 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:02.984474 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xn5c7\" (UniqueName: \"kubernetes.io/projected/0e431ffc-713b-41c8-bd1a-377110182a1c-kube-api-access-xn5c7\") pod \"0e431ffc-713b-41c8-bd1a-377110182a1c\" (UID: \"0e431ffc-713b-41c8-bd1a-377110182a1c\") " Jul 07 20:30:02.986579 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:02.986544 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e431ffc-713b-41c8-bd1a-377110182a1c-kube-api-access-xn5c7" (OuterVolumeSpecName: "kube-api-access-xn5c7") pod "0e431ffc-713b-41c8-bd1a-377110182a1c" (UID: "0e431ffc-713b-41c8-bd1a-377110182a1c"). InnerVolumeSpecName "kube-api-access-xn5c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 20:30:03.085921 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:03.085878 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn5c7\" (UniqueName: \"kubernetes.io/projected/0e431ffc-713b-41c8-bd1a-377110182a1c-kube-api-access-xn5c7\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:30:03.738633 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:03.738593 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724270-qh4mr" event={"ID":"0e431ffc-713b-41c8-bd1a-377110182a1c","Type":"ContainerDied","Data":"5da01e23335adf2640fc147992a736c50aef5719b306a6e968170e837b21a8d2"} Jul 07 20:30:03.738633 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:03.738631 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724270-qh4mr" Jul 07 20:30:03.738871 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:03.738645 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5da01e23335adf2640fc147992a736c50aef5719b306a6e968170e837b21a8d2" Jul 07 20:30:11.356004 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:11.355969 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8"] Jul 07 20:30:11.356388 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:11.356374 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e431ffc-713b-41c8-bd1a-377110182a1c" containerName="cleanup" Jul 07 20:30:11.356456 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:11.356390 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e431ffc-713b-41c8-bd1a-377110182a1c" containerName="cleanup" Jul 07 20:30:11.356503 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:11.356493 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e431ffc-713b-41c8-bd1a-377110182a1c" containerName="cleanup" Jul 07 20:30:11.359677 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:11.359658 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8" Jul 07 20:30:11.362176 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:11.362155 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-a3bfa138-serving-cert\"" Jul 07 20:30:11.368890 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:11.368865 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8"] Jul 07 20:30:11.464046 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:11.464004 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/489deaf4-07e8-434a-b25e-2ffc199f166d-maas-api-tls\") pod \"maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8\" (UID: \"489deaf4-07e8-434a-b25e-2ffc199f166d\") " pod="opendatahub/maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8" Jul 07 20:30:11.464046 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:11.464043 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hkkd\" (UniqueName: \"kubernetes.io/projected/489deaf4-07e8-434a-b25e-2ffc199f166d-kube-api-access-2hkkd\") pod \"maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8\" (UID: \"489deaf4-07e8-434a-b25e-2ffc199f166d\") " pod="opendatahub/maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8" Jul 07 20:30:11.464250 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:11.464146 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/489deaf4-07e8-434a-b25e-2ffc199f166d-openshift-service-ca\") pod \"maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8\" (UID: \"489deaf4-07e8-434a-b25e-2ffc199f166d\") " pod="opendatahub/maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8" Jul 07 20:30:11.565352 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:11.565320 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/489deaf4-07e8-434a-b25e-2ffc199f166d-maas-api-tls\") pod \"maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8\" (UID: \"489deaf4-07e8-434a-b25e-2ffc199f166d\") " pod="opendatahub/maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8" Jul 07 20:30:11.565352 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:11.565352 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hkkd\" (UniqueName: \"kubernetes.io/projected/489deaf4-07e8-434a-b25e-2ffc199f166d-kube-api-access-2hkkd\") pod \"maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8\" (UID: \"489deaf4-07e8-434a-b25e-2ffc199f166d\") " pod="opendatahub/maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8" Jul 07 20:30:11.565565 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:11.565418 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/489deaf4-07e8-434a-b25e-2ffc199f166d-openshift-service-ca\") pod \"maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8\" (UID: \"489deaf4-07e8-434a-b25e-2ffc199f166d\") " pod="opendatahub/maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8" Jul 07 20:30:11.565565 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:30:11.565494 2565 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-a3bfa138-serving-cert: secret "maas-api-e2e-mt-a3bfa138-serving-cert" not found Jul 07 20:30:11.565631 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:30:11.565567 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/489deaf4-07e8-434a-b25e-2ffc199f166d-maas-api-tls podName:489deaf4-07e8-434a-b25e-2ffc199f166d nodeName:}" failed. No retries permitted until 2026-07-07 20:30:12.065551899 +0000 UTC m=+1732.220013829 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/489deaf4-07e8-434a-b25e-2ffc199f166d-maas-api-tls") pod "maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8" (UID: "489deaf4-07e8-434a-b25e-2ffc199f166d") : secret "maas-api-e2e-mt-a3bfa138-serving-cert" not found Jul 07 20:30:11.566033 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:11.566015 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/489deaf4-07e8-434a-b25e-2ffc199f166d-openshift-service-ca\") pod \"maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8\" (UID: \"489deaf4-07e8-434a-b25e-2ffc199f166d\") " pod="opendatahub/maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8" Jul 07 20:30:11.574018 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:11.573989 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hkkd\" (UniqueName: \"kubernetes.io/projected/489deaf4-07e8-434a-b25e-2ffc199f166d-kube-api-access-2hkkd\") pod \"maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8\" (UID: \"489deaf4-07e8-434a-b25e-2ffc199f166d\") " pod="opendatahub/maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8" Jul 07 20:30:12.070241 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:12.070207 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/489deaf4-07e8-434a-b25e-2ffc199f166d-maas-api-tls\") pod \"maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8\" (UID: \"489deaf4-07e8-434a-b25e-2ffc199f166d\") " pod="opendatahub/maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8" Jul 07 20:30:12.072670 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:12.072640 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/489deaf4-07e8-434a-b25e-2ffc199f166d-maas-api-tls\") pod \"maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8\" (UID: \"489deaf4-07e8-434a-b25e-2ffc199f166d\") " pod="opendatahub/maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8" Jul 07 20:30:12.271531 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:12.271497 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8" Jul 07 20:30:12.410101 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:12.410075 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8"] Jul 07 20:30:12.412551 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:30:12.412523 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod489deaf4_07e8_434a_b25e_2ffc199f166d.slice/crio-883f8ea8460cb64027dcaad3aaf3d628b1a8cf4ef60bb72a1f3252789f46df2e WatchSource:0}: Error finding container 883f8ea8460cb64027dcaad3aaf3d628b1a8cf4ef60bb72a1f3252789f46df2e: Status 404 returned error can't find the container with id 883f8ea8460cb64027dcaad3aaf3d628b1a8cf4ef60bb72a1f3252789f46df2e Jul 07 20:30:12.782648 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:12.782609 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8" event={"ID":"489deaf4-07e8-434a-b25e-2ffc199f166d","Type":"ContainerStarted","Data":"883f8ea8460cb64027dcaad3aaf3d628b1a8cf4ef60bb72a1f3252789f46df2e"} Jul 07 20:30:13.788706 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:13.788670 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8" event={"ID":"489deaf4-07e8-434a-b25e-2ffc199f166d","Type":"ContainerStarted","Data":"4aa9a115501522ab1ebc1d5e15e0dc89d4b3224d7ca23ffd12b678d6ac90b004"} Jul 07 20:30:13.789096 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:13.788762 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8" Jul 07 20:30:13.809380 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:13.809331 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8" podStartSLOduration=2.30970472 podStartE2EDuration="2.809314789s" totalImagesPullingTime="499.610069ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:30:13.807788067 +0000 UTC m=+1733.962250023" watchObservedRunningTime="2026-07-07 20:30:13.809314789 +0000 UTC m=+1733.963776737" Jul 07 20:30:19.798692 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:19.798663 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8" Jul 07 20:30:28.687873 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:28.687837 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv"] Jul 07 20:30:28.691349 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:28.691327 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" Jul 07 20:30:28.695118 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:28.695091 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a3bfa138\"/\"kube-root-ca.crt\"" Jul 07 20:30:28.695118 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:28.695107 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a3bfa138\"/\"default-dockercfg-vdfs9\"" Jul 07 20:30:28.695372 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:28.695100 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a3bfa138\"/\"e2e-lifecycle-model-a3bfa138-kserve-self-signed-certs\"" Jul 07 20:30:28.695372 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:28.695210 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a3bfa138\"/\"openshift-service-ca.crt\"" Jul 07 20:30:28.702649 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:28.702624 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv"] Jul 07 20:30:28.826517 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:28.826481 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a51071fc-cf78-4749-828f-b7b7c8c5919f-dshm\") pod \"e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv\" (UID: \"a51071fc-cf78-4749-828f-b7b7c8c5919f\") " pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" Jul 07 20:30:28.826694 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:28.826531 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a51071fc-cf78-4749-828f-b7b7c8c5919f-model-cache\") pod \"e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv\" (UID: \"a51071fc-cf78-4749-828f-b7b7c8c5919f\") " pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" Jul 07 20:30:28.826694 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:28.826647 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a51071fc-cf78-4749-828f-b7b7c8c5919f-home\") pod \"e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv\" (UID: \"a51071fc-cf78-4749-828f-b7b7c8c5919f\") " pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" Jul 07 20:30:28.826694 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:28.826674 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7g9c\" (UniqueName: \"kubernetes.io/projected/a51071fc-cf78-4749-828f-b7b7c8c5919f-kube-api-access-p7g9c\") pod \"e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv\" (UID: \"a51071fc-cf78-4749-828f-b7b7c8c5919f\") " pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" Jul 07 20:30:28.826794 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:28.826713 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a51071fc-cf78-4749-828f-b7b7c8c5919f-tls-certs\") pod \"e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv\" (UID: \"a51071fc-cf78-4749-828f-b7b7c8c5919f\") " pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" Jul 07 20:30:28.826794 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:28.826758 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a51071fc-cf78-4749-828f-b7b7c8c5919f-tmp-dir\") pod \"e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv\" (UID: \"a51071fc-cf78-4749-828f-b7b7c8c5919f\") " pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" Jul 07 20:30:28.927359 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:28.927320 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a51071fc-cf78-4749-828f-b7b7c8c5919f-home\") pod \"e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv\" (UID: \"a51071fc-cf78-4749-828f-b7b7c8c5919f\") " pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" Jul 07 20:30:28.927359 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:28.927356 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7g9c\" (UniqueName: \"kubernetes.io/projected/a51071fc-cf78-4749-828f-b7b7c8c5919f-kube-api-access-p7g9c\") pod \"e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv\" (UID: \"a51071fc-cf78-4749-828f-b7b7c8c5919f\") " pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" Jul 07 20:30:28.927620 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:28.927388 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a51071fc-cf78-4749-828f-b7b7c8c5919f-tls-certs\") pod \"e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv\" (UID: \"a51071fc-cf78-4749-828f-b7b7c8c5919f\") " pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" Jul 07 20:30:28.927620 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:28.927424 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a51071fc-cf78-4749-828f-b7b7c8c5919f-tmp-dir\") pod \"e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv\" (UID: \"a51071fc-cf78-4749-828f-b7b7c8c5919f\") " pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" Jul 07 20:30:28.927620 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:28.927464 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a51071fc-cf78-4749-828f-b7b7c8c5919f-dshm\") pod \"e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv\" (UID: \"a51071fc-cf78-4749-828f-b7b7c8c5919f\") " pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" Jul 07 20:30:28.927620 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:28.927489 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a51071fc-cf78-4749-828f-b7b7c8c5919f-model-cache\") pod \"e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv\" (UID: \"a51071fc-cf78-4749-828f-b7b7c8c5919f\") " pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" Jul 07 20:30:28.927839 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:28.927794 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a51071fc-cf78-4749-828f-b7b7c8c5919f-home\") pod \"e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv\" (UID: \"a51071fc-cf78-4749-828f-b7b7c8c5919f\") " pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" Jul 07 20:30:28.927881 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:28.927846 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a51071fc-cf78-4749-828f-b7b7c8c5919f-tmp-dir\") pod \"e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv\" (UID: \"a51071fc-cf78-4749-828f-b7b7c8c5919f\") " pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" Jul 07 20:30:28.927946 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:28.927928 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a51071fc-cf78-4749-828f-b7b7c8c5919f-model-cache\") pod \"e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv\" (UID: \"a51071fc-cf78-4749-828f-b7b7c8c5919f\") " pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" Jul 07 20:30:28.929677 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:28.929654 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a51071fc-cf78-4749-828f-b7b7c8c5919f-dshm\") pod \"e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv\" (UID: \"a51071fc-cf78-4749-828f-b7b7c8c5919f\") " pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" Jul 07 20:30:28.929821 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:28.929802 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a51071fc-cf78-4749-828f-b7b7c8c5919f-tls-certs\") pod \"e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv\" (UID: \"a51071fc-cf78-4749-828f-b7b7c8c5919f\") " pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" Jul 07 20:30:28.937150 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:28.937120 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7g9c\" (UniqueName: \"kubernetes.io/projected/a51071fc-cf78-4749-828f-b7b7c8c5919f-kube-api-access-p7g9c\") pod \"e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv\" (UID: \"a51071fc-cf78-4749-828f-b7b7c8c5919f\") " pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" Jul 07 20:30:29.002286 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:29.002194 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" Jul 07 20:30:29.154763 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:29.154734 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv"] Jul 07 20:30:29.157153 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:30:29.157121 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda51071fc_cf78_4749_828f_b7b7c8c5919f.slice/crio-be655f430d528e188b94c16b15bf6333d9f886a20d52409688c232bcd01f6e5d WatchSource:0}: Error finding container be655f430d528e188b94c16b15bf6333d9f886a20d52409688c232bcd01f6e5d: Status 404 returned error can't find the container with id be655f430d528e188b94c16b15bf6333d9f886a20d52409688c232bcd01f6e5d Jul 07 20:30:29.867929 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:29.867897 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" event={"ID":"a51071fc-cf78-4749-828f-b7b7c8c5919f","Type":"ContainerStarted","Data":"17ae37e198aa114287081fa0f12184addabde75cc549d30c366671f89b011424"} Jul 07 20:30:29.868309 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:29.867938 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" event={"ID":"a51071fc-cf78-4749-828f-b7b7c8c5919f","Type":"ContainerStarted","Data":"be655f430d528e188b94c16b15bf6333d9f886a20d52409688c232bcd01f6e5d"} Jul 07 20:30:29.888002 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:29.887950 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" podStartSLOduration=1.887933343 podStartE2EDuration="1.887933343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:30:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:30:29.88684286 +0000 UTC m=+1750.041304812" watchObservedRunningTime="2026-07-07 20:30:29.887933343 +0000 UTC m=+1750.042395291" Jul 07 20:30:39.003298 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:39.003253 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" Jul 07 20:30:39.003739 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:39.003314 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" Jul 07 20:30:39.015927 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:39.015899 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" Jul 07 20:30:39.927621 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:39.927589 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" Jul 07 20:30:48.707139 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:48.707103 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8"] Jul 07 20:30:48.707566 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:48.707322 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8" podUID="489deaf4-07e8-434a-b25e-2ffc199f166d" containerName="maas-api" containerID="cri-o://4aa9a115501522ab1ebc1d5e15e0dc89d4b3224d7ca23ffd12b678d6ac90b004" gracePeriod=30 Jul 07 20:30:48.952646 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:48.952613 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8" Jul 07 20:30:48.961907 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:48.961819 2565 generic.go:356] "Generic (PLEG): container finished" podID="489deaf4-07e8-434a-b25e-2ffc199f166d" containerID="4aa9a115501522ab1ebc1d5e15e0dc89d4b3224d7ca23ffd12b678d6ac90b004" exitCode=0 Jul 07 20:30:48.961907 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:48.961854 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8" Jul 07 20:30:48.961907 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:48.961890 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8" event={"ID":"489deaf4-07e8-434a-b25e-2ffc199f166d","Type":"ContainerDied","Data":"4aa9a115501522ab1ebc1d5e15e0dc89d4b3224d7ca23ffd12b678d6ac90b004"} Jul 07 20:30:48.962156 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:48.961947 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8" event={"ID":"489deaf4-07e8-434a-b25e-2ffc199f166d","Type":"ContainerDied","Data":"883f8ea8460cb64027dcaad3aaf3d628b1a8cf4ef60bb72a1f3252789f46df2e"} Jul 07 20:30:48.962156 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:48.961976 2565 scope.go:117] "RemoveContainer" containerID="4aa9a115501522ab1ebc1d5e15e0dc89d4b3224d7ca23ffd12b678d6ac90b004" Jul 07 20:30:48.970521 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:48.970498 2565 scope.go:117] "RemoveContainer" containerID="4aa9a115501522ab1ebc1d5e15e0dc89d4b3224d7ca23ffd12b678d6ac90b004" Jul 07 20:30:48.970816 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:30:48.970799 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aa9a115501522ab1ebc1d5e15e0dc89d4b3224d7ca23ffd12b678d6ac90b004\": container with ID starting with 4aa9a115501522ab1ebc1d5e15e0dc89d4b3224d7ca23ffd12b678d6ac90b004 not found: ID does not exist" containerID="4aa9a115501522ab1ebc1d5e15e0dc89d4b3224d7ca23ffd12b678d6ac90b004" Jul 07 20:30:48.970884 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:48.970827 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa9a115501522ab1ebc1d5e15e0dc89d4b3224d7ca23ffd12b678d6ac90b004"} err="failed to get container status \"4aa9a115501522ab1ebc1d5e15e0dc89d4b3224d7ca23ffd12b678d6ac90b004\": rpc error: code = NotFound desc = could not find container \"4aa9a115501522ab1ebc1d5e15e0dc89d4b3224d7ca23ffd12b678d6ac90b004\": container with ID starting with 4aa9a115501522ab1ebc1d5e15e0dc89d4b3224d7ca23ffd12b678d6ac90b004 not found: ID does not exist" Jul 07 20:30:49.027574 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:49.027539 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/489deaf4-07e8-434a-b25e-2ffc199f166d-openshift-service-ca\") pod \"489deaf4-07e8-434a-b25e-2ffc199f166d\" (UID: \"489deaf4-07e8-434a-b25e-2ffc199f166d\") " Jul 07 20:30:49.027748 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:49.027587 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/489deaf4-07e8-434a-b25e-2ffc199f166d-maas-api-tls\") pod \"489deaf4-07e8-434a-b25e-2ffc199f166d\" (UID: \"489deaf4-07e8-434a-b25e-2ffc199f166d\") " Jul 07 20:30:49.027748 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:49.027611 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hkkd\" (UniqueName: \"kubernetes.io/projected/489deaf4-07e8-434a-b25e-2ffc199f166d-kube-api-access-2hkkd\") pod \"489deaf4-07e8-434a-b25e-2ffc199f166d\" (UID: \"489deaf4-07e8-434a-b25e-2ffc199f166d\") " Jul 07 20:30:49.028062 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:49.027936 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/489deaf4-07e8-434a-b25e-2ffc199f166d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "489deaf4-07e8-434a-b25e-2ffc199f166d" (UID: "489deaf4-07e8-434a-b25e-2ffc199f166d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 20:30:49.029690 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:49.029658 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/489deaf4-07e8-434a-b25e-2ffc199f166d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "489deaf4-07e8-434a-b25e-2ffc199f166d" (UID: "489deaf4-07e8-434a-b25e-2ffc199f166d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:30:49.029828 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:49.029806 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/489deaf4-07e8-434a-b25e-2ffc199f166d-kube-api-access-2hkkd" (OuterVolumeSpecName: "kube-api-access-2hkkd") pod "489deaf4-07e8-434a-b25e-2ffc199f166d" (UID: "489deaf4-07e8-434a-b25e-2ffc199f166d"). InnerVolumeSpecName "kube-api-access-2hkkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 20:30:49.128655 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:49.128609 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/489deaf4-07e8-434a-b25e-2ffc199f166d-openshift-service-ca\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:30:49.128655 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:49.128650 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/489deaf4-07e8-434a-b25e-2ffc199f166d-maas-api-tls\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:30:49.128655 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:49.128666 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hkkd\" (UniqueName: \"kubernetes.io/projected/489deaf4-07e8-434a-b25e-2ffc199f166d-kube-api-access-2hkkd\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:30:49.285316 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:49.285277 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8"] Jul 07 20:30:49.287104 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:49.287067 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-a3bfa138-c9758fcfb-kzms8"] Jul 07 20:30:49.685325 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:49.685287 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="489deaf4-07e8-434a-b25e-2ffc199f166d" path="/var/lib/kubelet/pods/489deaf4-07e8-434a-b25e-2ffc199f166d/volumes" Jul 07 20:30:59.489694 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:30:59.489658 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv"] Jul 07 20:31:14.510502 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:31:14.510392 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" podUID="a51071fc-cf78-4749-828f-b7b7c8c5919f" containerName="main" containerID="cri-o://17ae37e198aa114287081fa0f12184addabde75cc549d30c366671f89b011424" gracePeriod=45 Jul 07 20:31:14.763023 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:31:14.762950 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" Jul 07 20:31:14.873631 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:31:14.873573 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a51071fc-cf78-4749-828f-b7b7c8c5919f-home\") pod \"a51071fc-cf78-4749-828f-b7b7c8c5919f\" (UID: \"a51071fc-cf78-4749-828f-b7b7c8c5919f\") " Jul 07 20:31:14.873821 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:31:14.873709 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7g9c\" (UniqueName: \"kubernetes.io/projected/a51071fc-cf78-4749-828f-b7b7c8c5919f-kube-api-access-p7g9c\") pod \"a51071fc-cf78-4749-828f-b7b7c8c5919f\" (UID: \"a51071fc-cf78-4749-828f-b7b7c8c5919f\") " Jul 07 20:31:14.873821 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:31:14.873746 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a51071fc-cf78-4749-828f-b7b7c8c5919f-dshm\") pod \"a51071fc-cf78-4749-828f-b7b7c8c5919f\" (UID: \"a51071fc-cf78-4749-828f-b7b7c8c5919f\") " Jul 07 20:31:14.873821 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:31:14.873788 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a51071fc-cf78-4749-828f-b7b7c8c5919f-tls-certs\") pod \"a51071fc-cf78-4749-828f-b7b7c8c5919f\" (UID: \"a51071fc-cf78-4749-828f-b7b7c8c5919f\") " Jul 07 20:31:14.873979 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:31:14.873850 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a51071fc-cf78-4749-828f-b7b7c8c5919f-tmp-dir\") pod \"a51071fc-cf78-4749-828f-b7b7c8c5919f\" (UID: \"a51071fc-cf78-4749-828f-b7b7c8c5919f\") " Jul 07 20:31:14.873979 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:31:14.873882 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51071fc-cf78-4749-828f-b7b7c8c5919f-home" (OuterVolumeSpecName: "home") pod "a51071fc-cf78-4749-828f-b7b7c8c5919f" (UID: "a51071fc-cf78-4749-828f-b7b7c8c5919f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:31:14.873979 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:31:14.873921 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a51071fc-cf78-4749-828f-b7b7c8c5919f-model-cache\") pod \"a51071fc-cf78-4749-828f-b7b7c8c5919f\" (UID: \"a51071fc-cf78-4749-828f-b7b7c8c5919f\") " Jul 07 20:31:14.874229 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:31:14.874199 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51071fc-cf78-4749-828f-b7b7c8c5919f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a51071fc-cf78-4749-828f-b7b7c8c5919f" (UID: "a51071fc-cf78-4749-828f-b7b7c8c5919f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:31:14.874229 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:31:14.874205 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51071fc-cf78-4749-828f-b7b7c8c5919f-model-cache" (OuterVolumeSpecName: "model-cache") pod "a51071fc-cf78-4749-828f-b7b7c8c5919f" (UID: "a51071fc-cf78-4749-828f-b7b7c8c5919f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:31:14.874365 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:31:14.874259 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a51071fc-cf78-4749-828f-b7b7c8c5919f-home\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:31:14.876046 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:31:14.876017 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51071fc-cf78-4749-828f-b7b7c8c5919f-dshm" (OuterVolumeSpecName: "dshm") pod "a51071fc-cf78-4749-828f-b7b7c8c5919f" (UID: "a51071fc-cf78-4749-828f-b7b7c8c5919f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:31:14.876146 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:31:14.876027 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a51071fc-cf78-4749-828f-b7b7c8c5919f-kube-api-access-p7g9c" (OuterVolumeSpecName: "kube-api-access-p7g9c") pod "a51071fc-cf78-4749-828f-b7b7c8c5919f" (UID: "a51071fc-cf78-4749-828f-b7b7c8c5919f"). InnerVolumeSpecName "kube-api-access-p7g9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 20:31:14.876146 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:31:14.876084 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a51071fc-cf78-4749-828f-b7b7c8c5919f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a51071fc-cf78-4749-828f-b7b7c8c5919f" (UID: "a51071fc-cf78-4749-828f-b7b7c8c5919f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:31:14.975737 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:31:14.975688 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a51071fc-cf78-4749-828f-b7b7c8c5919f-tmp-dir\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:31:14.975737 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:31:14.975731 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a51071fc-cf78-4749-828f-b7b7c8c5919f-model-cache\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:31:14.975737 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:31:14.975741 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7g9c\" (UniqueName: \"kubernetes.io/projected/a51071fc-cf78-4749-828f-b7b7c8c5919f-kube-api-access-p7g9c\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:31:14.975737 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:31:14.975750 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a51071fc-cf78-4749-828f-b7b7c8c5919f-dshm\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:31:14.975737 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:31:14.975759 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a51071fc-cf78-4749-828f-b7b7c8c5919f-tls-certs\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:31:15.096132 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:31:15.096035 2565 generic.go:356] "Generic (PLEG): container finished" podID="a51071fc-cf78-4749-828f-b7b7c8c5919f" containerID="17ae37e198aa114287081fa0f12184addabde75cc549d30c366671f89b011424" exitCode=0 Jul 07 20:31:15.096132 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:31:15.096088 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" Jul 07 20:31:15.096311 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:31:15.096092 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" event={"ID":"a51071fc-cf78-4749-828f-b7b7c8c5919f","Type":"ContainerDied","Data":"17ae37e198aa114287081fa0f12184addabde75cc549d30c366671f89b011424"} Jul 07 20:31:15.096311 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:31:15.096213 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv" event={"ID":"a51071fc-cf78-4749-828f-b7b7c8c5919f","Type":"ContainerDied","Data":"be655f430d528e188b94c16b15bf6333d9f886a20d52409688c232bcd01f6e5d"} Jul 07 20:31:15.096311 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:31:15.096237 2565 scope.go:117] "RemoveContainer" containerID="17ae37e198aa114287081fa0f12184addabde75cc549d30c366671f89b011424" Jul 07 20:31:15.106042 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:31:15.106023 2565 scope.go:117] "RemoveContainer" containerID="17ae37e198aa114287081fa0f12184addabde75cc549d30c366671f89b011424" Jul 07 20:31:15.106347 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:31:15.106327 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17ae37e198aa114287081fa0f12184addabde75cc549d30c366671f89b011424\": container with ID starting with 17ae37e198aa114287081fa0f12184addabde75cc549d30c366671f89b011424 not found: ID does not exist" containerID="17ae37e198aa114287081fa0f12184addabde75cc549d30c366671f89b011424" Jul 07 20:31:15.106394 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:31:15.106356 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17ae37e198aa114287081fa0f12184addabde75cc549d30c366671f89b011424"} err="failed to get container status \"17ae37e198aa114287081fa0f12184addabde75cc549d30c366671f89b011424\": rpc error: code = NotFound desc = could not find container \"17ae37e198aa114287081fa0f12184addabde75cc549d30c366671f89b011424\": container with ID starting with 17ae37e198aa114287081fa0f12184addabde75cc549d30c366671f89b011424 not found: ID does not exist" Jul 07 20:31:15.119894 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:31:15.119857 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv"] Jul 07 20:31:15.124147 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:31:15.124119 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-a3bfa138/e2e-lifecycle-model-a3bfa138-kserve-794955dccf-kk4qv"] Jul 07 20:31:15.684564 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:31:15.684529 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a51071fc-cf78-4749-828f-b7b7c8c5919f" path="/var/lib/kubelet/pods/a51071fc-cf78-4749-828f-b7b7c8c5919f/volumes" Jul 07 20:32:45.449025 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:45.448939 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q"] Jul 07 20:32:45.449502 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:45.449360 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="489deaf4-07e8-434a-b25e-2ffc199f166d" containerName="maas-api" Jul 07 20:32:45.449502 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:45.449372 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="489deaf4-07e8-434a-b25e-2ffc199f166d" containerName="maas-api" Jul 07 20:32:45.449502 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:45.449412 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a51071fc-cf78-4749-828f-b7b7c8c5919f" containerName="main" Jul 07 20:32:45.449502 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:45.449418 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a51071fc-cf78-4749-828f-b7b7c8c5919f" containerName="main" Jul 07 20:32:45.449502 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:45.449500 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a51071fc-cf78-4749-828f-b7b7c8c5919f" containerName="main" Jul 07 20:32:45.449691 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:45.449508 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="489deaf4-07e8-434a-b25e-2ffc199f166d" containerName="maas-api" Jul 07 20:32:45.452838 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:45.452818 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q" Jul 07 20:32:45.455229 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:45.455205 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-c6e6f8-serving-cert\"" Jul 07 20:32:45.466298 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:45.466267 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q"] Jul 07 20:32:45.582513 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:45.582474 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw59n\" (UniqueName: \"kubernetes.io/projected/4840bf4b-566e-4cd5-a9d6-0d8e52f9785c-kube-api-access-zw59n\") pod \"maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q\" (UID: \"4840bf4b-566e-4cd5-a9d6-0d8e52f9785c\") " pod="opendatahub/maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q" Jul 07 20:32:45.582676 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:45.582533 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4840bf4b-566e-4cd5-a9d6-0d8e52f9785c-maas-api-tls\") pod \"maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q\" (UID: \"4840bf4b-566e-4cd5-a9d6-0d8e52f9785c\") " pod="opendatahub/maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q" Jul 07 20:32:45.582676 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:45.582561 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4840bf4b-566e-4cd5-a9d6-0d8e52f9785c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q\" (UID: \"4840bf4b-566e-4cd5-a9d6-0d8e52f9785c\") " pod="opendatahub/maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q" Jul 07 20:32:45.683270 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:45.683219 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4840bf4b-566e-4cd5-a9d6-0d8e52f9785c-maas-api-tls\") pod \"maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q\" (UID: \"4840bf4b-566e-4cd5-a9d6-0d8e52f9785c\") " pod="opendatahub/maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q" Jul 07 20:32:45.683270 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:45.683256 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4840bf4b-566e-4cd5-a9d6-0d8e52f9785c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q\" (UID: \"4840bf4b-566e-4cd5-a9d6-0d8e52f9785c\") " pod="opendatahub/maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q" Jul 07 20:32:45.683567 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:45.683439 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw59n\" (UniqueName: \"kubernetes.io/projected/4840bf4b-566e-4cd5-a9d6-0d8e52f9785c-kube-api-access-zw59n\") pod \"maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q\" (UID: \"4840bf4b-566e-4cd5-a9d6-0d8e52f9785c\") " pod="opendatahub/maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q" Jul 07 20:32:45.684072 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:45.684050 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4840bf4b-566e-4cd5-a9d6-0d8e52f9785c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q\" (UID: \"4840bf4b-566e-4cd5-a9d6-0d8e52f9785c\") " pod="opendatahub/maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q" Jul 07 20:32:45.685826 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:45.685807 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4840bf4b-566e-4cd5-a9d6-0d8e52f9785c-maas-api-tls\") pod \"maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q\" (UID: \"4840bf4b-566e-4cd5-a9d6-0d8e52f9785c\") " pod="opendatahub/maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q" Jul 07 20:32:45.692176 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:45.692153 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw59n\" (UniqueName: \"kubernetes.io/projected/4840bf4b-566e-4cd5-a9d6-0d8e52f9785c-kube-api-access-zw59n\") pod \"maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q\" (UID: \"4840bf4b-566e-4cd5-a9d6-0d8e52f9785c\") " pod="opendatahub/maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q" Jul 07 20:32:45.763355 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:45.763264 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q" Jul 07 20:32:45.901776 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:45.901752 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q"] Jul 07 20:32:45.904593 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:32:45.904555 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4840bf4b_566e_4cd5_a9d6_0d8e52f9785c.slice/crio-404837c3d186feff87833153fd20af0da5a0eb9df5bb31c97bd9b4d1adda5869 WatchSource:0}: Error finding container 404837c3d186feff87833153fd20af0da5a0eb9df5bb31c97bd9b4d1adda5869: Status 404 returned error can't find the container with id 404837c3d186feff87833153fd20af0da5a0eb9df5bb31c97bd9b4d1adda5869 Jul 07 20:32:45.906030 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:45.906011 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 20:32:46.544081 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:46.544051 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q" event={"ID":"4840bf4b-566e-4cd5-a9d6-0d8e52f9785c","Type":"ContainerStarted","Data":"89dda7ddffe612ba6fbfc1da0606525dc212155ee427928d8ae1b5aa21023f4a"} Jul 07 20:32:46.544490 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:46.544112 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q" event={"ID":"4840bf4b-566e-4cd5-a9d6-0d8e52f9785c","Type":"ContainerStarted","Data":"404837c3d186feff87833153fd20af0da5a0eb9df5bb31c97bd9b4d1adda5869"} Jul 07 20:32:46.544490 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:46.544153 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q" Jul 07 20:32:46.567930 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:46.567884 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q" podStartSLOduration=0.996333979 podStartE2EDuration="1.567869951s" totalImagesPullingTime="571.535972ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:32:46.565342366 +0000 UTC m=+1886.719804316" watchObservedRunningTime="2026-07-07 20:32:46.567869951 +0000 UTC m=+1886.722331899" Jul 07 20:32:52.556274 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:52.556245 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q" Jul 07 20:32:53.242704 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:53.242665 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv"] Jul 07 20:32:53.246156 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:53.246132 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv" Jul 07 20:32:53.248479 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:53.248450 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-080fc2-serving-cert\"" Jul 07 20:32:53.258922 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:53.258896 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv"] Jul 07 20:32:53.354957 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:53.354915 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07f34d29-d962-45a7-a459-33b07e3c303f-maas-api-tls\") pod \"maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv\" (UID: \"07f34d29-d962-45a7-a459-33b07e3c303f\") " pod="opendatahub/maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv" Jul 07 20:32:53.355149 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:53.354987 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f34d29-d962-45a7-a459-33b07e3c303f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv\" (UID: \"07f34d29-d962-45a7-a459-33b07e3c303f\") " pod="opendatahub/maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv" Jul 07 20:32:53.355149 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:53.355057 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhgtm\" (UniqueName: \"kubernetes.io/projected/07f34d29-d962-45a7-a459-33b07e3c303f-kube-api-access-rhgtm\") pod \"maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv\" (UID: \"07f34d29-d962-45a7-a459-33b07e3c303f\") " pod="opendatahub/maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv" Jul 07 20:32:53.455553 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:53.455512 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07f34d29-d962-45a7-a459-33b07e3c303f-maas-api-tls\") pod \"maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv\" (UID: \"07f34d29-d962-45a7-a459-33b07e3c303f\") " pod="opendatahub/maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv" Jul 07 20:32:53.455746 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:53.455587 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f34d29-d962-45a7-a459-33b07e3c303f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv\" (UID: \"07f34d29-d962-45a7-a459-33b07e3c303f\") " pod="opendatahub/maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv" Jul 07 20:32:53.455746 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:53.455617 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhgtm\" (UniqueName: \"kubernetes.io/projected/07f34d29-d962-45a7-a459-33b07e3c303f-kube-api-access-rhgtm\") pod \"maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv\" (UID: \"07f34d29-d962-45a7-a459-33b07e3c303f\") " pod="opendatahub/maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv" Jul 07 20:32:53.456249 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:53.456212 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f34d29-d962-45a7-a459-33b07e3c303f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv\" (UID: \"07f34d29-d962-45a7-a459-33b07e3c303f\") " pod="opendatahub/maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv" Jul 07 20:32:53.458088 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:53.458070 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07f34d29-d962-45a7-a459-33b07e3c303f-maas-api-tls\") pod \"maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv\" (UID: \"07f34d29-d962-45a7-a459-33b07e3c303f\") " pod="opendatahub/maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv" Jul 07 20:32:53.463813 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:53.463791 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhgtm\" (UniqueName: \"kubernetes.io/projected/07f34d29-d962-45a7-a459-33b07e3c303f-kube-api-access-rhgtm\") pod \"maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv\" (UID: \"07f34d29-d962-45a7-a459-33b07e3c303f\") " pod="opendatahub/maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv" Jul 07 20:32:53.559279 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:53.559184 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv" Jul 07 20:32:53.708013 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:53.707982 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv"] Jul 07 20:32:53.709920 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:32:53.709892 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f34d29_d962_45a7_a459_33b07e3c303f.slice/crio-69508c78857f8aabc76adb17126cf4a75acbc377876784587431a7ef420d365e WatchSource:0}: Error finding container 69508c78857f8aabc76adb17126cf4a75acbc377876784587431a7ef420d365e: Status 404 returned error can't find the container with id 69508c78857f8aabc76adb17126cf4a75acbc377876784587431a7ef420d365e Jul 07 20:32:54.590207 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:54.590106 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv" event={"ID":"07f34d29-d962-45a7-a459-33b07e3c303f","Type":"ContainerStarted","Data":"58b243f989df21ed0d5d06b7b327b6001dfed061a0dab2fc38898e0568f6416d"} Jul 07 20:32:54.590207 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:54.590167 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv" event={"ID":"07f34d29-d962-45a7-a459-33b07e3c303f","Type":"ContainerStarted","Data":"69508c78857f8aabc76adb17126cf4a75acbc377876784587431a7ef420d365e"} Jul 07 20:32:54.590207 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:54.590193 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv" Jul 07 20:32:54.609810 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:32:54.609746 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv" podStartSLOduration=1.139482396 podStartE2EDuration="1.609723387s" totalImagesPullingTime="470.240991ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:32:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:32:54.608392178 +0000 UTC m=+1894.762854128" watchObservedRunningTime="2026-07-07 20:32:54.609723387 +0000 UTC m=+1894.764185336" Jul 07 20:33:00.600305 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:00.600271 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv" Jul 07 20:33:03.787735 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:03.787698 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz"] Jul 07 20:33:03.791468 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:03.791448 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" Jul 07 20:33:03.793758 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:03.793733 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c6e6f8\"/\"openshift-service-ca.crt\"" Jul 07 20:33:03.793876 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:03.793741 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c6e6f8\"/\"default-dockercfg-gdhdc\"" Jul 07 20:33:03.794819 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:03.794797 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c6e6f8\"/\"kube-root-ca.crt\"" Jul 07 20:33:03.794913 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:03.794809 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c6e6f8\"/\"test-model-c6e6f8-kserve-self-signed-certs\"" Jul 07 20:33:03.801434 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:03.801394 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz"] Jul 07 20:33:03.955731 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:03.955687 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66330ba9-cbe0-4300-8062-10629bea3b21-home\") pod \"test-model-c6e6f8-kserve-85959b88fc-hhnqz\" (UID: \"66330ba9-cbe0-4300-8062-10629bea3b21\") " pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" Jul 07 20:33:03.955902 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:03.955738 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhsct\" (UniqueName: \"kubernetes.io/projected/66330ba9-cbe0-4300-8062-10629bea3b21-kube-api-access-zhsct\") pod \"test-model-c6e6f8-kserve-85959b88fc-hhnqz\" (UID: \"66330ba9-cbe0-4300-8062-10629bea3b21\") " pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" Jul 07 20:33:03.955902 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:03.955771 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66330ba9-cbe0-4300-8062-10629bea3b21-dshm\") pod \"test-model-c6e6f8-kserve-85959b88fc-hhnqz\" (UID: \"66330ba9-cbe0-4300-8062-10629bea3b21\") " pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" Jul 07 20:33:03.955902 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:03.955796 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66330ba9-cbe0-4300-8062-10629bea3b21-model-cache\") pod \"test-model-c6e6f8-kserve-85959b88fc-hhnqz\" (UID: \"66330ba9-cbe0-4300-8062-10629bea3b21\") " pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" Jul 07 20:33:03.955902 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:03.955844 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66330ba9-cbe0-4300-8062-10629bea3b21-tls-certs\") pod \"test-model-c6e6f8-kserve-85959b88fc-hhnqz\" (UID: \"66330ba9-cbe0-4300-8062-10629bea3b21\") " pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" Jul 07 20:33:03.956065 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:03.955907 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66330ba9-cbe0-4300-8062-10629bea3b21-tmp-dir\") pod \"test-model-c6e6f8-kserve-85959b88fc-hhnqz\" (UID: \"66330ba9-cbe0-4300-8062-10629bea3b21\") " pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" Jul 07 20:33:04.056945 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:04.056844 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66330ba9-cbe0-4300-8062-10629bea3b21-tmp-dir\") pod \"test-model-c6e6f8-kserve-85959b88fc-hhnqz\" (UID: \"66330ba9-cbe0-4300-8062-10629bea3b21\") " pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" Jul 07 20:33:04.056945 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:04.056921 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66330ba9-cbe0-4300-8062-10629bea3b21-home\") pod \"test-model-c6e6f8-kserve-85959b88fc-hhnqz\" (UID: \"66330ba9-cbe0-4300-8062-10629bea3b21\") " pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" Jul 07 20:33:04.057198 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:04.056963 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhsct\" (UniqueName: \"kubernetes.io/projected/66330ba9-cbe0-4300-8062-10629bea3b21-kube-api-access-zhsct\") pod \"test-model-c6e6f8-kserve-85959b88fc-hhnqz\" (UID: \"66330ba9-cbe0-4300-8062-10629bea3b21\") " pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" Jul 07 20:33:04.057198 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:04.056986 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66330ba9-cbe0-4300-8062-10629bea3b21-dshm\") pod \"test-model-c6e6f8-kserve-85959b88fc-hhnqz\" (UID: \"66330ba9-cbe0-4300-8062-10629bea3b21\") " pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" Jul 07 20:33:04.057198 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:04.057006 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66330ba9-cbe0-4300-8062-10629bea3b21-model-cache\") pod \"test-model-c6e6f8-kserve-85959b88fc-hhnqz\" (UID: \"66330ba9-cbe0-4300-8062-10629bea3b21\") " pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" Jul 07 20:33:04.057198 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:04.057049 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66330ba9-cbe0-4300-8062-10629bea3b21-tls-certs\") pod \"test-model-c6e6f8-kserve-85959b88fc-hhnqz\" (UID: \"66330ba9-cbe0-4300-8062-10629bea3b21\") " pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" Jul 07 20:33:04.057386 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:04.057275 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66330ba9-cbe0-4300-8062-10629bea3b21-home\") pod \"test-model-c6e6f8-kserve-85959b88fc-hhnqz\" (UID: \"66330ba9-cbe0-4300-8062-10629bea3b21\") " pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" Jul 07 20:33:04.057386 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:04.057321 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66330ba9-cbe0-4300-8062-10629bea3b21-tmp-dir\") pod \"test-model-c6e6f8-kserve-85959b88fc-hhnqz\" (UID: \"66330ba9-cbe0-4300-8062-10629bea3b21\") " pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" Jul 07 20:33:04.057512 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:04.057466 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66330ba9-cbe0-4300-8062-10629bea3b21-model-cache\") pod \"test-model-c6e6f8-kserve-85959b88fc-hhnqz\" (UID: \"66330ba9-cbe0-4300-8062-10629bea3b21\") " pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" Jul 07 20:33:04.059379 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:04.059351 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66330ba9-cbe0-4300-8062-10629bea3b21-dshm\") pod \"test-model-c6e6f8-kserve-85959b88fc-hhnqz\" (UID: \"66330ba9-cbe0-4300-8062-10629bea3b21\") " pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" Jul 07 20:33:04.059569 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:04.059553 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66330ba9-cbe0-4300-8062-10629bea3b21-tls-certs\") pod \"test-model-c6e6f8-kserve-85959b88fc-hhnqz\" (UID: \"66330ba9-cbe0-4300-8062-10629bea3b21\") " pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" Jul 07 20:33:04.066778 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:04.066746 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhsct\" (UniqueName: \"kubernetes.io/projected/66330ba9-cbe0-4300-8062-10629bea3b21-kube-api-access-zhsct\") pod \"test-model-c6e6f8-kserve-85959b88fc-hhnqz\" (UID: \"66330ba9-cbe0-4300-8062-10629bea3b21\") " pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" Jul 07 20:33:04.102571 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:04.102528 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" Jul 07 20:33:04.247229 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:04.247202 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz"] Jul 07 20:33:04.250098 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:33:04.250060 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66330ba9_cbe0_4300_8062_10629bea3b21.slice/crio-c036ba164d0d6fcbfe7c8b5b1c52728406dec9f34c0cac1d88790251e79ffa9c WatchSource:0}: Error finding container c036ba164d0d6fcbfe7c8b5b1c52728406dec9f34c0cac1d88790251e79ffa9c: Status 404 returned error can't find the container with id c036ba164d0d6fcbfe7c8b5b1c52728406dec9f34c0cac1d88790251e79ffa9c Jul 07 20:33:04.640908 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:04.640868 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" event={"ID":"66330ba9-cbe0-4300-8062-10629bea3b21","Type":"ContainerStarted","Data":"7e4f73bd16f7742df8e1e30395bf8b1e2ffb0ef5bdd6971a2a5afb4a13c1cb72"} Jul 07 20:33:04.640908 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:04.640911 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" event={"ID":"66330ba9-cbe0-4300-8062-10629bea3b21","Type":"ContainerStarted","Data":"c036ba164d0d6fcbfe7c8b5b1c52728406dec9f34c0cac1d88790251e79ffa9c"} Jul 07 20:33:04.659667 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:04.659612 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" podStartSLOduration=1.6595949060000001 podStartE2EDuration="1.659594906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:33:04.658512465 +0000 UTC m=+1904.812974413" watchObservedRunningTime="2026-07-07 20:33:04.659594906 +0000 UTC m=+1904.814056855" Jul 07 20:33:14.103179 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:14.103134 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" Jul 07 20:33:14.103640 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:14.103195 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" Jul 07 20:33:14.115876 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:14.115844 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" Jul 07 20:33:14.702572 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:33:14.702539 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" Jul 07 20:34:11.969162 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:11.969082 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz"] Jul 07 20:34:13.816242 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:13.816210 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q"] Jul 07 20:34:13.816668 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:13.816439 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q" podUID="4840bf4b-566e-4cd5-a9d6-0d8e52f9785c" containerName="maas-api" containerID="cri-o://89dda7ddffe612ba6fbfc1da0606525dc212155ee427928d8ae1b5aa21023f4a" gracePeriod=30 Jul 07 20:34:13.991479 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:13.991453 2565 generic.go:356] "Generic (PLEG): container finished" podID="4840bf4b-566e-4cd5-a9d6-0d8e52f9785c" containerID="89dda7ddffe612ba6fbfc1da0606525dc212155ee427928d8ae1b5aa21023f4a" exitCode=0 Jul 07 20:34:13.991615 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:13.991521 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q" event={"ID":"4840bf4b-566e-4cd5-a9d6-0d8e52f9785c","Type":"ContainerDied","Data":"89dda7ddffe612ba6fbfc1da0606525dc212155ee427928d8ae1b5aa21023f4a"} Jul 07 20:34:14.061600 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:14.061577 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q" Jul 07 20:34:14.152171 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:14.152147 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4840bf4b-566e-4cd5-a9d6-0d8e52f9785c-openshift-service-ca\") pod \"4840bf4b-566e-4cd5-a9d6-0d8e52f9785c\" (UID: \"4840bf4b-566e-4cd5-a9d6-0d8e52f9785c\") " Jul 07 20:34:14.152347 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:14.152190 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zw59n\" (UniqueName: \"kubernetes.io/projected/4840bf4b-566e-4cd5-a9d6-0d8e52f9785c-kube-api-access-zw59n\") pod \"4840bf4b-566e-4cd5-a9d6-0d8e52f9785c\" (UID: \"4840bf4b-566e-4cd5-a9d6-0d8e52f9785c\") " Jul 07 20:34:14.152347 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:14.152291 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4840bf4b-566e-4cd5-a9d6-0d8e52f9785c-maas-api-tls\") pod \"4840bf4b-566e-4cd5-a9d6-0d8e52f9785c\" (UID: \"4840bf4b-566e-4cd5-a9d6-0d8e52f9785c\") " Jul 07 20:34:14.152577 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:14.152553 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4840bf4b-566e-4cd5-a9d6-0d8e52f9785c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4840bf4b-566e-4cd5-a9d6-0d8e52f9785c" (UID: "4840bf4b-566e-4cd5-a9d6-0d8e52f9785c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 20:34:14.154285 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:14.154258 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4840bf4b-566e-4cd5-a9d6-0d8e52f9785c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4840bf4b-566e-4cd5-a9d6-0d8e52f9785c" (UID: "4840bf4b-566e-4cd5-a9d6-0d8e52f9785c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:34:14.154383 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:14.154314 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4840bf4b-566e-4cd5-a9d6-0d8e52f9785c-kube-api-access-zw59n" (OuterVolumeSpecName: "kube-api-access-zw59n") pod "4840bf4b-566e-4cd5-a9d6-0d8e52f9785c" (UID: "4840bf4b-566e-4cd5-a9d6-0d8e52f9785c"). InnerVolumeSpecName "kube-api-access-zw59n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 20:34:14.253199 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:14.253167 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4840bf4b-566e-4cd5-a9d6-0d8e52f9785c-maas-api-tls\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:34:14.253199 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:14.253193 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4840bf4b-566e-4cd5-a9d6-0d8e52f9785c-openshift-service-ca\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:34:14.253199 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:14.253204 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw59n\" (UniqueName: \"kubernetes.io/projected/4840bf4b-566e-4cd5-a9d6-0d8e52f9785c-kube-api-access-zw59n\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:34:14.997855 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:14.997824 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q" Jul 07 20:34:14.998320 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:14.997868 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q" event={"ID":"4840bf4b-566e-4cd5-a9d6-0d8e52f9785c","Type":"ContainerDied","Data":"404837c3d186feff87833153fd20af0da5a0eb9df5bb31c97bd9b4d1adda5869"} Jul 07 20:34:14.998320 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:14.997930 2565 scope.go:117] "RemoveContainer" containerID="89dda7ddffe612ba6fbfc1da0606525dc212155ee427928d8ae1b5aa21023f4a" Jul 07 20:34:15.020035 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:15.020010 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q"] Jul 07 20:34:15.024006 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:15.023982 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-c6e6f8-854c5d4d49-btb2q"] Jul 07 20:34:15.684729 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:15.684698 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4840bf4b-566e-4cd5-a9d6-0d8e52f9785c" path="/var/lib/kubelet/pods/4840bf4b-566e-4cd5-a9d6-0d8e52f9785c/volumes" Jul 07 20:34:26.989679 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:26.989616 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" podUID="66330ba9-cbe0-4300-8062-10629bea3b21" containerName="main" containerID="cri-o://7e4f73bd16f7742df8e1e30395bf8b1e2ffb0ef5bdd6971a2a5afb4a13c1cb72" gracePeriod=45 Jul 07 20:34:27.245220 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:27.245153 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" Jul 07 20:34:27.274154 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:27.274117 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhsct\" (UniqueName: \"kubernetes.io/projected/66330ba9-cbe0-4300-8062-10629bea3b21-kube-api-access-zhsct\") pod \"66330ba9-cbe0-4300-8062-10629bea3b21\" (UID: \"66330ba9-cbe0-4300-8062-10629bea3b21\") " Jul 07 20:34:27.274327 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:27.274171 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66330ba9-cbe0-4300-8062-10629bea3b21-tls-certs\") pod \"66330ba9-cbe0-4300-8062-10629bea3b21\" (UID: \"66330ba9-cbe0-4300-8062-10629bea3b21\") " Jul 07 20:34:27.274327 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:27.274271 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66330ba9-cbe0-4300-8062-10629bea3b21-tmp-dir\") pod \"66330ba9-cbe0-4300-8062-10629bea3b21\" (UID: \"66330ba9-cbe0-4300-8062-10629bea3b21\") " Jul 07 20:34:27.274327 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:27.274303 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66330ba9-cbe0-4300-8062-10629bea3b21-dshm\") pod \"66330ba9-cbe0-4300-8062-10629bea3b21\" (UID: \"66330ba9-cbe0-4300-8062-10629bea3b21\") " Jul 07 20:34:27.274544 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:27.274341 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66330ba9-cbe0-4300-8062-10629bea3b21-model-cache\") pod \"66330ba9-cbe0-4300-8062-10629bea3b21\" (UID: \"66330ba9-cbe0-4300-8062-10629bea3b21\") " Jul 07 20:34:27.274544 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:27.274435 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66330ba9-cbe0-4300-8062-10629bea3b21-home\") pod \"66330ba9-cbe0-4300-8062-10629bea3b21\" (UID: \"66330ba9-cbe0-4300-8062-10629bea3b21\") " Jul 07 20:34:27.274655 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:27.274595 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66330ba9-cbe0-4300-8062-10629bea3b21-model-cache" (OuterVolumeSpecName: "model-cache") pod "66330ba9-cbe0-4300-8062-10629bea3b21" (UID: "66330ba9-cbe0-4300-8062-10629bea3b21"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:34:27.274655 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:27.274606 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66330ba9-cbe0-4300-8062-10629bea3b21-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "66330ba9-cbe0-4300-8062-10629bea3b21" (UID: "66330ba9-cbe0-4300-8062-10629bea3b21"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:34:27.274824 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:27.274807 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66330ba9-cbe0-4300-8062-10629bea3b21-model-cache\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:34:27.274870 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:27.274829 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66330ba9-cbe0-4300-8062-10629bea3b21-home" (OuterVolumeSpecName: "home") pod "66330ba9-cbe0-4300-8062-10629bea3b21" (UID: "66330ba9-cbe0-4300-8062-10629bea3b21"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:34:27.274870 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:27.274858 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66330ba9-cbe0-4300-8062-10629bea3b21-tmp-dir\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:34:27.276519 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:27.276490 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66330ba9-cbe0-4300-8062-10629bea3b21-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "66330ba9-cbe0-4300-8062-10629bea3b21" (UID: "66330ba9-cbe0-4300-8062-10629bea3b21"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:34:27.276784 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:27.276761 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66330ba9-cbe0-4300-8062-10629bea3b21-dshm" (OuterVolumeSpecName: "dshm") pod "66330ba9-cbe0-4300-8062-10629bea3b21" (UID: "66330ba9-cbe0-4300-8062-10629bea3b21"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:34:27.276856 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:27.276800 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66330ba9-cbe0-4300-8062-10629bea3b21-kube-api-access-zhsct" (OuterVolumeSpecName: "kube-api-access-zhsct") pod "66330ba9-cbe0-4300-8062-10629bea3b21" (UID: "66330ba9-cbe0-4300-8062-10629bea3b21"). InnerVolumeSpecName "kube-api-access-zhsct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 20:34:27.375947 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:27.375912 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66330ba9-cbe0-4300-8062-10629bea3b21-home\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:34:27.375947 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:27.375943 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhsct\" (UniqueName: \"kubernetes.io/projected/66330ba9-cbe0-4300-8062-10629bea3b21-kube-api-access-zhsct\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:34:27.375947 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:27.375955 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66330ba9-cbe0-4300-8062-10629bea3b21-tls-certs\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:34:27.376184 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:27.375964 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66330ba9-cbe0-4300-8062-10629bea3b21-dshm\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:34:28.063963 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:28.063872 2565 generic.go:356] "Generic (PLEG): container finished" podID="66330ba9-cbe0-4300-8062-10629bea3b21" containerID="7e4f73bd16f7742df8e1e30395bf8b1e2ffb0ef5bdd6971a2a5afb4a13c1cb72" exitCode=0 Jul 07 20:34:28.063963 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:28.063917 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" Jul 07 20:34:28.063963 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:28.063949 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" event={"ID":"66330ba9-cbe0-4300-8062-10629bea3b21","Type":"ContainerDied","Data":"7e4f73bd16f7742df8e1e30395bf8b1e2ffb0ef5bdd6971a2a5afb4a13c1cb72"} Jul 07 20:34:28.064501 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:28.063981 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz" event={"ID":"66330ba9-cbe0-4300-8062-10629bea3b21","Type":"ContainerDied","Data":"c036ba164d0d6fcbfe7c8b5b1c52728406dec9f34c0cac1d88790251e79ffa9c"} Jul 07 20:34:28.064501 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:28.063998 2565 scope.go:117] "RemoveContainer" containerID="7e4f73bd16f7742df8e1e30395bf8b1e2ffb0ef5bdd6971a2a5afb4a13c1cb72" Jul 07 20:34:28.072805 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:28.072784 2565 scope.go:117] "RemoveContainer" containerID="7e4f73bd16f7742df8e1e30395bf8b1e2ffb0ef5bdd6971a2a5afb4a13c1cb72" Jul 07 20:34:28.073138 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:34:28.073115 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e4f73bd16f7742df8e1e30395bf8b1e2ffb0ef5bdd6971a2a5afb4a13c1cb72\": container with ID starting with 7e4f73bd16f7742df8e1e30395bf8b1e2ffb0ef5bdd6971a2a5afb4a13c1cb72 not found: ID does not exist" containerID="7e4f73bd16f7742df8e1e30395bf8b1e2ffb0ef5bdd6971a2a5afb4a13c1cb72" Jul 07 20:34:28.073196 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:28.073146 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e4f73bd16f7742df8e1e30395bf8b1e2ffb0ef5bdd6971a2a5afb4a13c1cb72"} err="failed to get container status \"7e4f73bd16f7742df8e1e30395bf8b1e2ffb0ef5bdd6971a2a5afb4a13c1cb72\": rpc error: code = NotFound desc = could not find container \"7e4f73bd16f7742df8e1e30395bf8b1e2ffb0ef5bdd6971a2a5afb4a13c1cb72\": container with ID starting with 7e4f73bd16f7742df8e1e30395bf8b1e2ffb0ef5bdd6971a2a5afb4a13c1cb72 not found: ID does not exist" Jul 07 20:34:28.082142 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:28.082086 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz"] Jul 07 20:34:28.084071 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:28.084046 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c6e6f8/test-model-c6e6f8-kserve-85959b88fc-hhnqz"] Jul 07 20:34:29.684511 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:29.684476 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66330ba9-cbe0-4300-8062-10629bea3b21" path="/var/lib/kubelet/pods/66330ba9-cbe0-4300-8062-10629bea3b21/volumes" Jul 07 20:34:57.297227 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:57.297195 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv"] Jul 07 20:34:57.297800 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:57.297394 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv" podUID="07f34d29-d962-45a7-a459-33b07e3c303f" containerName="maas-api" containerID="cri-o://58b243f989df21ed0d5d06b7b327b6001dfed061a0dab2fc38898e0568f6416d" gracePeriod=30 Jul 07 20:34:57.742206 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:57.742179 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv" Jul 07 20:34:57.870829 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:57.870745 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhgtm\" (UniqueName: \"kubernetes.io/projected/07f34d29-d962-45a7-a459-33b07e3c303f-kube-api-access-rhgtm\") pod \"07f34d29-d962-45a7-a459-33b07e3c303f\" (UID: \"07f34d29-d962-45a7-a459-33b07e3c303f\") " Jul 07 20:34:57.870829 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:57.870808 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07f34d29-d962-45a7-a459-33b07e3c303f-maas-api-tls\") pod \"07f34d29-d962-45a7-a459-33b07e3c303f\" (UID: \"07f34d29-d962-45a7-a459-33b07e3c303f\") " Jul 07 20:34:57.871037 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:57.870843 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f34d29-d962-45a7-a459-33b07e3c303f-openshift-service-ca\") pod \"07f34d29-d962-45a7-a459-33b07e3c303f\" (UID: \"07f34d29-d962-45a7-a459-33b07e3c303f\") " Jul 07 20:34:57.871252 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:57.871222 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07f34d29-d962-45a7-a459-33b07e3c303f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "07f34d29-d962-45a7-a459-33b07e3c303f" (UID: "07f34d29-d962-45a7-a459-33b07e3c303f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 20:34:57.872811 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:57.872787 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f34d29-d962-45a7-a459-33b07e3c303f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "07f34d29-d962-45a7-a459-33b07e3c303f" (UID: "07f34d29-d962-45a7-a459-33b07e3c303f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:34:57.872909 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:57.872866 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f34d29-d962-45a7-a459-33b07e3c303f-kube-api-access-rhgtm" (OuterVolumeSpecName: "kube-api-access-rhgtm") pod "07f34d29-d962-45a7-a459-33b07e3c303f" (UID: "07f34d29-d962-45a7-a459-33b07e3c303f"). InnerVolumeSpecName "kube-api-access-rhgtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 20:34:57.971786 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:57.971746 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhgtm\" (UniqueName: \"kubernetes.io/projected/07f34d29-d962-45a7-a459-33b07e3c303f-kube-api-access-rhgtm\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:34:57.971786 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:57.971784 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07f34d29-d962-45a7-a459-33b07e3c303f-maas-api-tls\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:34:57.971786 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:57.971794 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f34d29-d962-45a7-a459-33b07e3c303f-openshift-service-ca\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:34:58.208574 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:58.208534 2565 generic.go:356] "Generic (PLEG): container finished" podID="07f34d29-d962-45a7-a459-33b07e3c303f" containerID="58b243f989df21ed0d5d06b7b327b6001dfed061a0dab2fc38898e0568f6416d" exitCode=0 Jul 07 20:34:58.208751 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:58.208583 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv" Jul 07 20:34:58.208751 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:58.208605 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv" event={"ID":"07f34d29-d962-45a7-a459-33b07e3c303f","Type":"ContainerDied","Data":"58b243f989df21ed0d5d06b7b327b6001dfed061a0dab2fc38898e0568f6416d"} Jul 07 20:34:58.208751 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:58.208657 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv" event={"ID":"07f34d29-d962-45a7-a459-33b07e3c303f","Type":"ContainerDied","Data":"69508c78857f8aabc76adb17126cf4a75acbc377876784587431a7ef420d365e"} Jul 07 20:34:58.208751 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:58.208685 2565 scope.go:117] "RemoveContainer" containerID="58b243f989df21ed0d5d06b7b327b6001dfed061a0dab2fc38898e0568f6416d" Jul 07 20:34:58.217677 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:58.217659 2565 scope.go:117] "RemoveContainer" containerID="58b243f989df21ed0d5d06b7b327b6001dfed061a0dab2fc38898e0568f6416d" Jul 07 20:34:58.217913 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:34:58.217894 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58b243f989df21ed0d5d06b7b327b6001dfed061a0dab2fc38898e0568f6416d\": container with ID starting with 58b243f989df21ed0d5d06b7b327b6001dfed061a0dab2fc38898e0568f6416d not found: ID does not exist" containerID="58b243f989df21ed0d5d06b7b327b6001dfed061a0dab2fc38898e0568f6416d" Jul 07 20:34:58.217959 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:58.217922 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58b243f989df21ed0d5d06b7b327b6001dfed061a0dab2fc38898e0568f6416d"} err="failed to get container status \"58b243f989df21ed0d5d06b7b327b6001dfed061a0dab2fc38898e0568f6416d\": rpc error: code = NotFound desc = could not find container \"58b243f989df21ed0d5d06b7b327b6001dfed061a0dab2fc38898e0568f6416d\": container with ID starting with 58b243f989df21ed0d5d06b7b327b6001dfed061a0dab2fc38898e0568f6416d not found: ID does not exist" Jul 07 20:34:58.230134 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:58.230098 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv"] Jul 07 20:34:58.234007 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:58.233978 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-080fc2-78cdfdc7b5-c6hhv"] Jul 07 20:34:59.685559 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:34:59.685522 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07f34d29-d962-45a7-a459-33b07e3c303f" path="/var/lib/kubelet/pods/07f34d29-d962-45a7-a459-33b07e3c303f/volumes" Jul 07 20:35:04.350145 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.350112 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b"] Jul 07 20:35:04.350618 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.350520 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4840bf4b-566e-4cd5-a9d6-0d8e52f9785c" containerName="maas-api" Jul 07 20:35:04.350618 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.350532 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4840bf4b-566e-4cd5-a9d6-0d8e52f9785c" containerName="maas-api" Jul 07 20:35:04.350618 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.350540 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66330ba9-cbe0-4300-8062-10629bea3b21" containerName="main" Jul 07 20:35:04.350618 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.350546 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="66330ba9-cbe0-4300-8062-10629bea3b21" containerName="main" Jul 07 20:35:04.350618 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.350561 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07f34d29-d962-45a7-a459-33b07e3c303f" containerName="maas-api" Jul 07 20:35:04.350618 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.350566 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="07f34d29-d962-45a7-a459-33b07e3c303f" containerName="maas-api" Jul 07 20:35:04.350618 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.350621 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="07f34d29-d962-45a7-a459-33b07e3c303f" containerName="maas-api" Jul 07 20:35:04.350839 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.350633 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4840bf4b-566e-4cd5-a9d6-0d8e52f9785c" containerName="maas-api" Jul 07 20:35:04.350839 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.350639 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="66330ba9-cbe0-4300-8062-10629bea3b21" containerName="main" Jul 07 20:35:04.355241 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.355221 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:04.358108 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.358089 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 20:35:04.358210 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.358193 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-7db6f1-openshift-default-dockercfg-kssh8\"" Jul 07 20:35:04.365486 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.365453 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b"] Jul 07 20:35:04.425978 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.425930 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frjkf\" (UniqueName: \"kubernetes.io/projected/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-kube-api-access-frjkf\") pod \"e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:04.425978 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.425981 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-workload-certs\") pod \"e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:04.426213 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.426041 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-workload-socket\") pod \"e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:04.426213 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.426098 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-istio-data\") pod \"e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:04.426213 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.426138 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-istio-token\") pod \"e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:04.426213 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.426159 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-istio-envoy\") pod \"e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:04.426344 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.426221 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-istiod-ca-cert\") pod \"e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:04.426344 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.426263 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-istio-podinfo\") pod \"e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:04.426344 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.426295 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-credential-socket\") pod \"e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:04.526812 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.526777 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-istio-token\") pod \"e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:04.526812 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.526819 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-istio-envoy\") pod \"e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:04.527044 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.526861 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-istiod-ca-cert\") pod \"e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:04.527044 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.526886 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-istio-podinfo\") pod \"e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:04.527044 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.526913 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-credential-socket\") pod \"e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:04.527044 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.526947 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frjkf\" (UniqueName: \"kubernetes.io/projected/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-kube-api-access-frjkf\") pod \"e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:04.527261 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.527235 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-workload-certs\") pod \"e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:04.527319 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.527305 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-workload-socket\") pod \"e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:04.527374 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.527354 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-credential-socket\") pod \"e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:04.527374 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.527365 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-istio-data\") pod \"e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:04.527636 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.527611 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-workload-certs\") pod \"e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:04.527770 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.527669 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-istio-data\") pod \"e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:04.527770 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.527696 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-workload-socket\") pod \"e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:04.527770 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.527756 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-istiod-ca-cert\") pod \"e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:04.529917 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.529894 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-istio-envoy\") pod \"e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:04.530088 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.530073 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-istio-podinfo\") pod \"e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:04.537657 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.537630 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-istio-token\") pod \"e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:04.538862 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.538844 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frjkf\" (UniqueName: \"kubernetes.io/projected/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-kube-api-access-frjkf\") pod \"e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:04.667748 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.667705 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:04.808858 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:04.808831 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b"] Jul 07 20:35:04.810885 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:35:04.810857 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0c09fb_bcaf_431e_9394_b0d2a8ab491e.slice/crio-5f7036f604df16dda12f30e2c3fdf7471ef9708bf2e2bb1c3ff2aa210992135f WatchSource:0}: Error finding container 5f7036f604df16dda12f30e2c3fdf7471ef9708bf2e2bb1c3ff2aa210992135f: Status 404 returned error can't find the container with id 5f7036f604df16dda12f30e2c3fdf7471ef9708bf2e2bb1c3ff2aa210992135f Jul 07 20:35:05.246775 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:05.246742 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" event={"ID":"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e","Type":"ContainerStarted","Data":"5f7036f604df16dda12f30e2c3fdf7471ef9708bf2e2bb1c3ff2aa210992135f"} Jul 07 20:35:08.073512 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:08.073472 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 20:35:08.073807 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:08.073548 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 20:35:08.073807 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:08.073594 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 20:35:08.265659 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:08.265621 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" event={"ID":"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e","Type":"ContainerStarted","Data":"d41ceb1396c5018e5a8f013294bebe916980814c16b934353384dd85898d620e"} Jul 07 20:35:08.290783 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:08.290720 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" podStartSLOduration=1.030262295 podStartE2EDuration="4.290694243s" totalImagesPullingTime="3.260431948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:35:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:35:08.287842029 +0000 UTC m=+2028.442303978" watchObservedRunningTime="2026-07-07 20:35:08.290694243 +0000 UTC m=+2028.445156193" Jul 07 20:35:08.668605 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:08.668547 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:09.672940 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:09.672910 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:10.276184 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:10.276147 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:10.276962 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:10.276946 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:10.388996 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:10.388963 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v"] Jul 07 20:35:10.392711 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:10.392689 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v" Jul 07 20:35:10.395519 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:10.395498 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-7db6f1-serving-cert\"" Jul 07 20:35:10.401935 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:10.401910 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v"] Jul 07 20:35:10.486771 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:10.486736 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5984bb72-9790-4d20-a1db-0e838d861b3c-openshift-service-ca\") pod \"maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v\" (UID: \"5984bb72-9790-4d20-a1db-0e838d861b3c\") " pod="opendatahub/maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v" Jul 07 20:35:10.486941 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:10.486800 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4hc5\" (UniqueName: \"kubernetes.io/projected/5984bb72-9790-4d20-a1db-0e838d861b3c-kube-api-access-n4hc5\") pod \"maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v\" (UID: \"5984bb72-9790-4d20-a1db-0e838d861b3c\") " pod="opendatahub/maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v" Jul 07 20:35:10.486941 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:10.486915 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5984bb72-9790-4d20-a1db-0e838d861b3c-maas-api-tls\") pod \"maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v\" (UID: \"5984bb72-9790-4d20-a1db-0e838d861b3c\") " pod="opendatahub/maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v" Jul 07 20:35:10.587938 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:10.587843 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4hc5\" (UniqueName: \"kubernetes.io/projected/5984bb72-9790-4d20-a1db-0e838d861b3c-kube-api-access-n4hc5\") pod \"maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v\" (UID: \"5984bb72-9790-4d20-a1db-0e838d861b3c\") " pod="opendatahub/maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v" Jul 07 20:35:10.587938 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:10.587897 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5984bb72-9790-4d20-a1db-0e838d861b3c-maas-api-tls\") pod \"maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v\" (UID: \"5984bb72-9790-4d20-a1db-0e838d861b3c\") " pod="opendatahub/maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v" Jul 07 20:35:10.588149 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:10.588063 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5984bb72-9790-4d20-a1db-0e838d861b3c-openshift-service-ca\") pod \"maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v\" (UID: \"5984bb72-9790-4d20-a1db-0e838d861b3c\") " pod="opendatahub/maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v" Jul 07 20:35:10.588779 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:10.588762 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5984bb72-9790-4d20-a1db-0e838d861b3c-openshift-service-ca\") pod \"maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v\" (UID: \"5984bb72-9790-4d20-a1db-0e838d861b3c\") " pod="opendatahub/maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v" Jul 07 20:35:10.590356 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:10.590338 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5984bb72-9790-4d20-a1db-0e838d861b3c-maas-api-tls\") pod \"maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v\" (UID: \"5984bb72-9790-4d20-a1db-0e838d861b3c\") " pod="opendatahub/maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v" Jul 07 20:35:10.595894 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:10.595871 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4hc5\" (UniqueName: \"kubernetes.io/projected/5984bb72-9790-4d20-a1db-0e838d861b3c-kube-api-access-n4hc5\") pod \"maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v\" (UID: \"5984bb72-9790-4d20-a1db-0e838d861b3c\") " pod="opendatahub/maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v" Jul 07 20:35:10.704188 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:10.704157 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v" Jul 07 20:35:10.857292 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:10.857250 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v"] Jul 07 20:35:10.859792 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:35:10.859763 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5984bb72_9790_4d20_a1db_0e838d861b3c.slice/crio-c1ee3f971a3e4a60db4043bbe432282d7be44dbf5c70bc52b6bb869feca712a1 WatchSource:0}: Error finding container c1ee3f971a3e4a60db4043bbe432282d7be44dbf5c70bc52b6bb869feca712a1: Status 404 returned error can't find the container with id c1ee3f971a3e4a60db4043bbe432282d7be44dbf5c70bc52b6bb869feca712a1 Jul 07 20:35:11.282160 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:11.282125 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v" event={"ID":"5984bb72-9790-4d20-a1db-0e838d861b3c","Type":"ContainerStarted","Data":"c1ee3f971a3e4a60db4043bbe432282d7be44dbf5c70bc52b6bb869feca712a1"} Jul 07 20:35:12.288723 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:12.288677 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v" event={"ID":"5984bb72-9790-4d20-a1db-0e838d861b3c","Type":"ContainerStarted","Data":"3f102cc80412cb6c6665e8f40ef7d7d6e23de7b0ed0fefdd78357a3879366f39"} Jul 07 20:35:12.309810 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:12.309760 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v" podStartSLOduration=1.819157067 podStartE2EDuration="2.309744171s" totalImagesPullingTime="490.587104ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:35:12.308331601 +0000 UTC m=+2032.462793551" watchObservedRunningTime="2026-07-07 20:35:12.309744171 +0000 UTC m=+2032.464206114" Jul 07 20:35:13.293590 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:13.293560 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v" Jul 07 20:35:19.303925 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:19.303891 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v" Jul 07 20:35:19.322929 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:19.322801 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4"] Jul 07 20:35:19.327142 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:19.327113 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4" Jul 07 20:35:19.329992 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:19.329854 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-c9e405-serving-cert\"" Jul 07 20:35:19.341600 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:19.340241 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4"] Jul 07 20:35:19.369695 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:19.369462 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f158d3ca-ff57-4007-8c86-8cf38c704001-openshift-service-ca\") pod \"maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4\" (UID: \"f158d3ca-ff57-4007-8c86-8cf38c704001\") " pod="opendatahub/maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4" Jul 07 20:35:19.369695 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:19.369507 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f158d3ca-ff57-4007-8c86-8cf38c704001-maas-api-tls\") pod \"maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4\" (UID: \"f158d3ca-ff57-4007-8c86-8cf38c704001\") " pod="opendatahub/maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4" Jul 07 20:35:19.369695 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:19.369551 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2kzq\" (UniqueName: \"kubernetes.io/projected/f158d3ca-ff57-4007-8c86-8cf38c704001-kube-api-access-d2kzq\") pod \"maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4\" (UID: \"f158d3ca-ff57-4007-8c86-8cf38c704001\") " pod="opendatahub/maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4" Jul 07 20:35:19.470875 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:19.470835 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2kzq\" (UniqueName: \"kubernetes.io/projected/f158d3ca-ff57-4007-8c86-8cf38c704001-kube-api-access-d2kzq\") pod \"maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4\" (UID: \"f158d3ca-ff57-4007-8c86-8cf38c704001\") " pod="opendatahub/maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4" Jul 07 20:35:19.471128 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:19.470948 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f158d3ca-ff57-4007-8c86-8cf38c704001-openshift-service-ca\") pod \"maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4\" (UID: \"f158d3ca-ff57-4007-8c86-8cf38c704001\") " pod="opendatahub/maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4" Jul 07 20:35:19.471128 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:19.470966 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f158d3ca-ff57-4007-8c86-8cf38c704001-maas-api-tls\") pod \"maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4\" (UID: \"f158d3ca-ff57-4007-8c86-8cf38c704001\") " pod="opendatahub/maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4" Jul 07 20:35:19.471128 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:35:19.471071 2565 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-c9e405-serving-cert: secret "maas-api-e2e-api-b-c9e405-serving-cert" not found Jul 07 20:35:19.471279 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:35:19.471131 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f158d3ca-ff57-4007-8c86-8cf38c704001-maas-api-tls podName:f158d3ca-ff57-4007-8c86-8cf38c704001 nodeName:}" failed. No retries permitted until 2026-07-07 20:35:19.971114008 +0000 UTC m=+2040.125575955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f158d3ca-ff57-4007-8c86-8cf38c704001-maas-api-tls") pod "maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4" (UID: "f158d3ca-ff57-4007-8c86-8cf38c704001") : secret "maas-api-e2e-api-b-c9e405-serving-cert" not found Jul 07 20:35:19.471652 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:19.471629 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f158d3ca-ff57-4007-8c86-8cf38c704001-openshift-service-ca\") pod \"maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4\" (UID: \"f158d3ca-ff57-4007-8c86-8cf38c704001\") " pod="opendatahub/maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4" Jul 07 20:35:19.482060 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:19.482032 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2kzq\" (UniqueName: \"kubernetes.io/projected/f158d3ca-ff57-4007-8c86-8cf38c704001-kube-api-access-d2kzq\") pod \"maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4\" (UID: \"f158d3ca-ff57-4007-8c86-8cf38c704001\") " pod="opendatahub/maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4" Jul 07 20:35:19.976788 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:19.976755 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f158d3ca-ff57-4007-8c86-8cf38c704001-maas-api-tls\") pod \"maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4\" (UID: \"f158d3ca-ff57-4007-8c86-8cf38c704001\") " pod="opendatahub/maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4" Jul 07 20:35:19.979490 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:19.979459 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f158d3ca-ff57-4007-8c86-8cf38c704001-maas-api-tls\") pod \"maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4\" (UID: \"f158d3ca-ff57-4007-8c86-8cf38c704001\") " pod="opendatahub/maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4" Jul 07 20:35:20.242337 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:20.242247 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4" Jul 07 20:35:20.383275 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:20.383251 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4"] Jul 07 20:35:20.385720 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:35:20.385690 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf158d3ca_ff57_4007_8c86_8cf38c704001.slice/crio-a22d1b2a50142bf944c036eff6b6d56d2499a93bea8e0f6a11b95da3b0c844af WatchSource:0}: Error finding container a22d1b2a50142bf944c036eff6b6d56d2499a93bea8e0f6a11b95da3b0c844af: Status 404 returned error can't find the container with id a22d1b2a50142bf944c036eff6b6d56d2499a93bea8e0f6a11b95da3b0c844af Jul 07 20:35:21.353192 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:21.353097 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4" event={"ID":"f158d3ca-ff57-4007-8c86-8cf38c704001","Type":"ContainerStarted","Data":"fb4f28f02dc423404232c71d99ecb4cecae7c78fd89945ca507d769f9da1a4aa"} Jul 07 20:35:21.353340 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:21.353256 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4" Jul 07 20:35:21.353340 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:21.353302 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4" event={"ID":"f158d3ca-ff57-4007-8c86-8cf38c704001","Type":"ContainerStarted","Data":"a22d1b2a50142bf944c036eff6b6d56d2499a93bea8e0f6a11b95da3b0c844af"} Jul 07 20:35:21.372199 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:21.372151 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4" podStartSLOduration=1.817261843 podStartE2EDuration="2.372136537s" totalImagesPullingTime="554.874694ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:35:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:35:21.370672934 +0000 UTC m=+2041.525134884" watchObservedRunningTime="2026-07-07 20:35:21.372136537 +0000 UTC m=+2041.526598514" Jul 07 20:35:22.359618 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:22.359583 2565 generic.go:356] "Generic (PLEG): container finished" podID="0b0c09fb-bcaf-431e-9394-b0d2a8ab491e" containerID="d41ceb1396c5018e5a8f013294bebe916980814c16b934353384dd85898d620e" exitCode=0 Jul 07 20:35:22.360075 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:22.359658 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" event={"ID":"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e","Type":"ContainerDied","Data":"d41ceb1396c5018e5a8f013294bebe916980814c16b934353384dd85898d620e"} Jul 07 20:35:22.360218 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:22.360200 2565 scope.go:117] "RemoveContainer" containerID="d41ceb1396c5018e5a8f013294bebe916980814c16b934353384dd85898d620e" Jul 07 20:35:22.362122 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:22.362094 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 20:35:22.362223 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:22.362155 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 20:35:22.362223 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:22.362186 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 20:35:23.366874 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:23.366831 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" event={"ID":"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e","Type":"ContainerStarted","Data":"39903aafbaa016d3a589316948afc13b362861e39061a9a4674046c36274291a"} Jul 07 20:35:23.669165 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:23.669127 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:23.669622 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:23.669591 2565 patch_prober.go:28] interesting pod/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 07 20:35:23.669819 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:23.669787 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" podUID="0b0c09fb-bcaf-431e-9394-b0d2a8ab491e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 07 20:35:24.678956 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:24.678916 2565 patch_prober.go:28] interesting pod/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 07 20:35:24.679579 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:24.679546 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" podUID="0b0c09fb-bcaf-431e-9394-b0d2a8ab491e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 07 20:35:25.277073 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:25.277027 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:25.668735 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:25.668694 2565 patch_prober.go:28] interesting pod/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 07 20:35:25.668954 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:25.668763 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" podUID="0b0c09fb-bcaf-431e-9394-b0d2a8ab491e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 07 20:35:26.668965 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:26.668924 2565 patch_prober.go:28] interesting pod/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 07 20:35:26.669529 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:26.668994 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" podUID="0b0c09fb-bcaf-431e-9394-b0d2a8ab491e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 07 20:35:27.365620 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:27.365591 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4" Jul 07 20:35:27.671968 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:27.671941 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:27.672877 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:27.672858 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:32.027021 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:32.026977 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v"] Jul 07 20:35:32.027498 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:32.027240 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v" podUID="5984bb72-9790-4d20-a1db-0e838d861b3c" containerName="maas-api" containerID="cri-o://3f102cc80412cb6c6665e8f40ef7d7d6e23de7b0ed0fefdd78357a3879366f39" gracePeriod=30 Jul 07 20:35:32.281522 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:32.281456 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v" Jul 07 20:35:32.401655 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:32.401619 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5984bb72-9790-4d20-a1db-0e838d861b3c-maas-api-tls\") pod \"5984bb72-9790-4d20-a1db-0e838d861b3c\" (UID: \"5984bb72-9790-4d20-a1db-0e838d861b3c\") " Jul 07 20:35:32.401836 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:32.401712 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4hc5\" (UniqueName: \"kubernetes.io/projected/5984bb72-9790-4d20-a1db-0e838d861b3c-kube-api-access-n4hc5\") pod \"5984bb72-9790-4d20-a1db-0e838d861b3c\" (UID: \"5984bb72-9790-4d20-a1db-0e838d861b3c\") " Jul 07 20:35:32.401836 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:32.401747 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5984bb72-9790-4d20-a1db-0e838d861b3c-openshift-service-ca\") pod \"5984bb72-9790-4d20-a1db-0e838d861b3c\" (UID: \"5984bb72-9790-4d20-a1db-0e838d861b3c\") " Jul 07 20:35:32.402140 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:32.402112 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5984bb72-9790-4d20-a1db-0e838d861b3c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5984bb72-9790-4d20-a1db-0e838d861b3c" (UID: "5984bb72-9790-4d20-a1db-0e838d861b3c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 20:35:32.403783 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:32.403761 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5984bb72-9790-4d20-a1db-0e838d861b3c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5984bb72-9790-4d20-a1db-0e838d861b3c" (UID: "5984bb72-9790-4d20-a1db-0e838d861b3c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:35:32.403867 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:32.403790 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5984bb72-9790-4d20-a1db-0e838d861b3c-kube-api-access-n4hc5" (OuterVolumeSpecName: "kube-api-access-n4hc5") pod "5984bb72-9790-4d20-a1db-0e838d861b3c" (UID: "5984bb72-9790-4d20-a1db-0e838d861b3c"). InnerVolumeSpecName "kube-api-access-n4hc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 20:35:32.424017 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:32.423987 2565 generic.go:356] "Generic (PLEG): container finished" podID="5984bb72-9790-4d20-a1db-0e838d861b3c" containerID="3f102cc80412cb6c6665e8f40ef7d7d6e23de7b0ed0fefdd78357a3879366f39" exitCode=0 Jul 07 20:35:32.424173 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:32.424032 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v" Jul 07 20:35:32.424173 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:32.424074 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v" event={"ID":"5984bb72-9790-4d20-a1db-0e838d861b3c","Type":"ContainerDied","Data":"3f102cc80412cb6c6665e8f40ef7d7d6e23de7b0ed0fefdd78357a3879366f39"} Jul 07 20:35:32.424173 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:32.424123 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v" event={"ID":"5984bb72-9790-4d20-a1db-0e838d861b3c","Type":"ContainerDied","Data":"c1ee3f971a3e4a60db4043bbe432282d7be44dbf5c70bc52b6bb869feca712a1"} Jul 07 20:35:32.424173 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:32.424145 2565 scope.go:117] "RemoveContainer" containerID="3f102cc80412cb6c6665e8f40ef7d7d6e23de7b0ed0fefdd78357a3879366f39" Jul 07 20:35:32.433081 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:32.433066 2565 scope.go:117] "RemoveContainer" containerID="3f102cc80412cb6c6665e8f40ef7d7d6e23de7b0ed0fefdd78357a3879366f39" Jul 07 20:35:32.433352 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:35:32.433333 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f102cc80412cb6c6665e8f40ef7d7d6e23de7b0ed0fefdd78357a3879366f39\": container with ID starting with 3f102cc80412cb6c6665e8f40ef7d7d6e23de7b0ed0fefdd78357a3879366f39 not found: ID does not exist" containerID="3f102cc80412cb6c6665e8f40ef7d7d6e23de7b0ed0fefdd78357a3879366f39" Jul 07 20:35:32.433449 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:32.433359 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f102cc80412cb6c6665e8f40ef7d7d6e23de7b0ed0fefdd78357a3879366f39"} err="failed to get container status \"3f102cc80412cb6c6665e8f40ef7d7d6e23de7b0ed0fefdd78357a3879366f39\": rpc error: code = NotFound desc = could not find container \"3f102cc80412cb6c6665e8f40ef7d7d6e23de7b0ed0fefdd78357a3879366f39\": container with ID starting with 3f102cc80412cb6c6665e8f40ef7d7d6e23de7b0ed0fefdd78357a3879366f39 not found: ID does not exist" Jul 07 20:35:32.446339 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:32.446310 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v"] Jul 07 20:35:32.452472 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:32.452447 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-7db6f1-867fc66bfb-86d9v"] Jul 07 20:35:32.502996 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:32.502970 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4hc5\" (UniqueName: \"kubernetes.io/projected/5984bb72-9790-4d20-a1db-0e838d861b3c-kube-api-access-n4hc5\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:35:32.502996 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:32.502995 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5984bb72-9790-4d20-a1db-0e838d861b3c-openshift-service-ca\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:35:32.502996 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:32.503006 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5984bb72-9790-4d20-a1db-0e838d861b3c-maas-api-tls\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:35:33.684943 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:33.684909 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5984bb72-9790-4d20-a1db-0e838d861b3c" path="/var/lib/kubelet/pods/5984bb72-9790-4d20-a1db-0e838d861b3c/volumes" Jul 07 20:35:39.263144 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:39.263112 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b"] Jul 07 20:35:39.263621 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:39.263335 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" podUID="0b0c09fb-bcaf-431e-9394-b0d2a8ab491e" containerName="istio-proxy" containerID="cri-o://39903aafbaa016d3a589316948afc13b362861e39061a9a4674046c36274291a" gracePeriod=30 Jul 07 20:35:40.520129 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:40.520099 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4"] Jul 07 20:35:40.520568 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:40.520305 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4" podUID="f158d3ca-ff57-4007-8c86-8cf38c704001" containerName="maas-api" containerID="cri-o://fb4f28f02dc423404232c71d99ecb4cecae7c78fd89945ca507d769f9da1a4aa" gracePeriod=30 Jul 07 20:35:40.774888 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:40.774825 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4" Jul 07 20:35:40.880574 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:40.880540 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f158d3ca-ff57-4007-8c86-8cf38c704001-maas-api-tls\") pod \"f158d3ca-ff57-4007-8c86-8cf38c704001\" (UID: \"f158d3ca-ff57-4007-8c86-8cf38c704001\") " Jul 07 20:35:40.880790 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:40.880595 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f158d3ca-ff57-4007-8c86-8cf38c704001-openshift-service-ca\") pod \"f158d3ca-ff57-4007-8c86-8cf38c704001\" (UID: \"f158d3ca-ff57-4007-8c86-8cf38c704001\") " Jul 07 20:35:40.880790 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:40.880674 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2kzq\" (UniqueName: \"kubernetes.io/projected/f158d3ca-ff57-4007-8c86-8cf38c704001-kube-api-access-d2kzq\") pod \"f158d3ca-ff57-4007-8c86-8cf38c704001\" (UID: \"f158d3ca-ff57-4007-8c86-8cf38c704001\") " Jul 07 20:35:40.880972 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:40.880943 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f158d3ca-ff57-4007-8c86-8cf38c704001-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f158d3ca-ff57-4007-8c86-8cf38c704001" (UID: "f158d3ca-ff57-4007-8c86-8cf38c704001"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 20:35:40.882723 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:40.882699 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f158d3ca-ff57-4007-8c86-8cf38c704001-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f158d3ca-ff57-4007-8c86-8cf38c704001" (UID: "f158d3ca-ff57-4007-8c86-8cf38c704001"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:35:40.882815 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:40.882790 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f158d3ca-ff57-4007-8c86-8cf38c704001-kube-api-access-d2kzq" (OuterVolumeSpecName: "kube-api-access-d2kzq") pod "f158d3ca-ff57-4007-8c86-8cf38c704001" (UID: "f158d3ca-ff57-4007-8c86-8cf38c704001"). InnerVolumeSpecName "kube-api-access-d2kzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 20:35:40.982379 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:40.982332 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f158d3ca-ff57-4007-8c86-8cf38c704001-openshift-service-ca\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:35:40.982379 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:40.982368 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2kzq\" (UniqueName: \"kubernetes.io/projected/f158d3ca-ff57-4007-8c86-8cf38c704001-kube-api-access-d2kzq\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:35:40.982379 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:40.982378 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f158d3ca-ff57-4007-8c86-8cf38c704001-maas-api-tls\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:35:41.471506 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:41.471474 2565 generic.go:356] "Generic (PLEG): container finished" podID="f158d3ca-ff57-4007-8c86-8cf38c704001" containerID="fb4f28f02dc423404232c71d99ecb4cecae7c78fd89945ca507d769f9da1a4aa" exitCode=0 Jul 07 20:35:41.471689 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:41.471526 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4" event={"ID":"f158d3ca-ff57-4007-8c86-8cf38c704001","Type":"ContainerDied","Data":"fb4f28f02dc423404232c71d99ecb4cecae7c78fd89945ca507d769f9da1a4aa"} Jul 07 20:35:41.471689 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:41.471534 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4" Jul 07 20:35:41.471689 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:41.471592 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4" event={"ID":"f158d3ca-ff57-4007-8c86-8cf38c704001","Type":"ContainerDied","Data":"a22d1b2a50142bf944c036eff6b6d56d2499a93bea8e0f6a11b95da3b0c844af"} Jul 07 20:35:41.471689 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:41.471614 2565 scope.go:117] "RemoveContainer" containerID="fb4f28f02dc423404232c71d99ecb4cecae7c78fd89945ca507d769f9da1a4aa" Jul 07 20:35:41.481261 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:41.481241 2565 scope.go:117] "RemoveContainer" containerID="fb4f28f02dc423404232c71d99ecb4cecae7c78fd89945ca507d769f9da1a4aa" Jul 07 20:35:41.481540 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:35:41.481521 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb4f28f02dc423404232c71d99ecb4cecae7c78fd89945ca507d769f9da1a4aa\": container with ID starting with fb4f28f02dc423404232c71d99ecb4cecae7c78fd89945ca507d769f9da1a4aa not found: ID does not exist" containerID="fb4f28f02dc423404232c71d99ecb4cecae7c78fd89945ca507d769f9da1a4aa" Jul 07 20:35:41.481595 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:41.481549 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4f28f02dc423404232c71d99ecb4cecae7c78fd89945ca507d769f9da1a4aa"} err="failed to get container status \"fb4f28f02dc423404232c71d99ecb4cecae7c78fd89945ca507d769f9da1a4aa\": rpc error: code = NotFound desc = could not find container \"fb4f28f02dc423404232c71d99ecb4cecae7c78fd89945ca507d769f9da1a4aa\": container with ID starting with fb4f28f02dc423404232c71d99ecb4cecae7c78fd89945ca507d769f9da1a4aa not found: ID does not exist" Jul 07 20:35:41.494751 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:41.494717 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4"] Jul 07 20:35:41.499008 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:41.498974 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-c9e405-5cb6d69fdf-cr6g4"] Jul 07 20:35:41.684678 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:41.684605 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f158d3ca-ff57-4007-8c86-8cf38c704001" path="/var/lib/kubelet/pods/f158d3ca-ff57-4007-8c86-8cf38c704001/volumes" Jul 07 20:35:42.674760 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:42.674725 2565 patch_prober.go:28] interesting pod/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 20:35:42.674942 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:42.674777 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" podUID="0b0c09fb-bcaf-431e-9394-b0d2a8ab491e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 20:35:44.490759 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:44.490720 2565 generic.go:356] "Generic (PLEG): container finished" podID="0b0c09fb-bcaf-431e-9394-b0d2a8ab491e" containerID="39903aafbaa016d3a589316948afc13b362861e39061a9a4674046c36274291a" exitCode=0 Jul 07 20:35:44.491117 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:44.490796 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" event={"ID":"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e","Type":"ContainerDied","Data":"39903aafbaa016d3a589316948afc13b362861e39061a9a4674046c36274291a"} Jul 07 20:35:44.491117 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:44.490881 2565 scope.go:117] "RemoveContainer" containerID="d41ceb1396c5018e5a8f013294bebe916980814c16b934353384dd85898d620e" Jul 07 20:35:44.523859 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:44.523812 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:44.719433 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:44.719370 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-istio-token\") pod \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " Jul 07 20:35:44.719433 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:44.719425 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-istio-podinfo\") pod \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " Jul 07 20:35:44.719663 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:44.719479 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-istio-envoy\") pod \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " Jul 07 20:35:44.719663 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:44.719543 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-credential-socket\") pod \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " Jul 07 20:35:44.719663 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:44.719575 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-workload-socket\") pod \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " Jul 07 20:35:44.719663 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:44.719622 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frjkf\" (UniqueName: \"kubernetes.io/projected/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-kube-api-access-frjkf\") pod \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " Jul 07 20:35:44.719663 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:44.719665 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-workload-certs\") pod \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " Jul 07 20:35:44.719919 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:44.719687 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-istio-data\") pod \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " Jul 07 20:35:44.719919 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:44.719709 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-istiod-ca-cert\") pod \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\" (UID: \"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e\") " Jul 07 20:35:44.719919 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:44.719867 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0b0c09fb-bcaf-431e-9394-b0d2a8ab491e" (UID: "0b0c09fb-bcaf-431e-9394-b0d2a8ab491e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:35:44.720077 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:44.719879 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0b0c09fb-bcaf-431e-9394-b0d2a8ab491e" (UID: "0b0c09fb-bcaf-431e-9394-b0d2a8ab491e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:35:44.720077 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:44.719973 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0b0c09fb-bcaf-431e-9394-b0d2a8ab491e" (UID: "0b0c09fb-bcaf-431e-9394-b0d2a8ab491e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:35:44.720346 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:44.720320 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0b0c09fb-bcaf-431e-9394-b0d2a8ab491e" (UID: "0b0c09fb-bcaf-431e-9394-b0d2a8ab491e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 20:35:44.720511 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:44.720491 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-workload-certs\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:35:44.720599 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:44.720517 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-istiod-ca-cert\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:35:44.720599 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:44.720531 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-credential-socket\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:35:44.720599 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:44.720547 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-workload-socket\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:35:44.720599 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:44.720493 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-istio-data" (OuterVolumeSpecName: "istio-data") pod "0b0c09fb-bcaf-431e-9394-b0d2a8ab491e" (UID: "0b0c09fb-bcaf-431e-9394-b0d2a8ab491e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:35:44.721937 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:44.721915 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0b0c09fb-bcaf-431e-9394-b0d2a8ab491e" (UID: "0b0c09fb-bcaf-431e-9394-b0d2a8ab491e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:35:44.722033 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:44.721942 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-kube-api-access-frjkf" (OuterVolumeSpecName: "kube-api-access-frjkf") pod "0b0c09fb-bcaf-431e-9394-b0d2a8ab491e" (UID: "0b0c09fb-bcaf-431e-9394-b0d2a8ab491e"). InnerVolumeSpecName "kube-api-access-frjkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 20:35:44.722033 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:44.722011 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0b0c09fb-bcaf-431e-9394-b0d2a8ab491e" (UID: "0b0c09fb-bcaf-431e-9394-b0d2a8ab491e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 20:35:44.722109 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:44.722034 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-istio-token" (OuterVolumeSpecName: "istio-token") pod "0b0c09fb-bcaf-431e-9394-b0d2a8ab491e" (UID: "0b0c09fb-bcaf-431e-9394-b0d2a8ab491e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 20:35:44.821740 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:44.821652 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-istio-data\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:35:44.821740 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:44.821682 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-istio-token\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:35:44.821740 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:44.821690 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-istio-podinfo\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:35:44.821740 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:44.821699 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-istio-envoy\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:35:44.821740 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:44.821707 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frjkf\" (UniqueName: \"kubernetes.io/projected/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e-kube-api-access-frjkf\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:35:45.496866 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:45.496841 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" Jul 07 20:35:45.497296 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:45.496889 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b" event={"ID":"0b0c09fb-bcaf-431e-9394-b0d2a8ab491e","Type":"ContainerDied","Data":"5f7036f604df16dda12f30e2c3fdf7471ef9708bf2e2bb1c3ff2aa210992135f"} Jul 07 20:35:45.497296 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:45.496953 2565 scope.go:117] "RemoveContainer" containerID="39903aafbaa016d3a589316948afc13b362861e39061a9a4674046c36274291a" Jul 07 20:35:45.519286 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:45.519261 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b"] Jul 07 20:35:45.523269 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:45.523246 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-7db6f1-openshift-default-65cb99c747-qxj4b"] Jul 07 20:35:45.685809 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:45.685771 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b0c09fb-bcaf-431e-9394-b0d2a8ab491e" path="/var/lib/kubelet/pods/0b0c09fb-bcaf-431e-9394-b0d2a8ab491e/volumes" Jul 07 20:35:47.651805 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.651769 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg"] Jul 07 20:35:47.652171 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.652158 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5984bb72-9790-4d20-a1db-0e838d861b3c" containerName="maas-api" Jul 07 20:35:47.652227 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.652172 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5984bb72-9790-4d20-a1db-0e838d861b3c" containerName="maas-api" Jul 07 20:35:47.652227 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.652183 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f158d3ca-ff57-4007-8c86-8cf38c704001" containerName="maas-api" Jul 07 20:35:47.652227 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.652189 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f158d3ca-ff57-4007-8c86-8cf38c704001" containerName="maas-api" Jul 07 20:35:47.652227 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.652208 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b0c09fb-bcaf-431e-9394-b0d2a8ab491e" containerName="istio-proxy" Jul 07 20:35:47.652227 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.652213 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b0c09fb-bcaf-431e-9394-b0d2a8ab491e" containerName="istio-proxy" Jul 07 20:35:47.652227 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.652221 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b0c09fb-bcaf-431e-9394-b0d2a8ab491e" containerName="istio-proxy" Jul 07 20:35:47.652227 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.652226 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b0c09fb-bcaf-431e-9394-b0d2a8ab491e" containerName="istio-proxy" Jul 07 20:35:47.652473 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.652284 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5984bb72-9790-4d20-a1db-0e838d861b3c" containerName="maas-api" Jul 07 20:35:47.652473 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.652292 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f158d3ca-ff57-4007-8c86-8cf38c704001" containerName="maas-api" Jul 07 20:35:47.652473 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.652297 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b0c09fb-bcaf-431e-9394-b0d2a8ab491e" containerName="istio-proxy" Jul 07 20:35:47.652473 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.652304 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b0c09fb-bcaf-431e-9394-b0d2a8ab491e" containerName="istio-proxy" Jul 07 20:35:47.655465 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.655446 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:47.657960 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.657941 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 20:35:47.658076 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.657964 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-e9f8cf-openshift-default-dockercfg-7cp2g\"" Jul 07 20:35:47.663419 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.663373 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg"] Jul 07 20:35:47.742804 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.742767 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09df4297-042d-4762-b66a-3dab2e909d45-istio-podinfo\") pod \"e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:47.742804 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.742806 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09df4297-042d-4762-b66a-3dab2e909d45-workload-socket\") pod \"e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:47.743037 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.742871 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09df4297-042d-4762-b66a-3dab2e909d45-istio-token\") pod \"e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:47.743037 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.742918 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09df4297-042d-4762-b66a-3dab2e909d45-credential-socket\") pod \"e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:47.743037 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.742963 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09df4297-042d-4762-b66a-3dab2e909d45-workload-certs\") pod \"e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:47.743037 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.743013 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09df4297-042d-4762-b66a-3dab2e909d45-istio-envoy\") pod \"e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:47.743200 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.743041 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09df4297-042d-4762-b66a-3dab2e909d45-istio-data\") pod \"e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:47.743200 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.743186 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09df4297-042d-4762-b66a-3dab2e909d45-istiod-ca-cert\") pod \"e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:47.743268 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.743233 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gc69\" (UniqueName: \"kubernetes.io/projected/09df4297-042d-4762-b66a-3dab2e909d45-kube-api-access-6gc69\") pod \"e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:47.844245 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.844213 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09df4297-042d-4762-b66a-3dab2e909d45-istio-token\") pod \"e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:47.844245 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.844250 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09df4297-042d-4762-b66a-3dab2e909d45-credential-socket\") pod \"e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:47.844515 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.844266 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09df4297-042d-4762-b66a-3dab2e909d45-workload-certs\") pod \"e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:47.844515 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.844281 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09df4297-042d-4762-b66a-3dab2e909d45-istio-envoy\") pod \"e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:47.844515 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.844297 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09df4297-042d-4762-b66a-3dab2e909d45-istio-data\") pod \"e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:47.844515 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.844366 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09df4297-042d-4762-b66a-3dab2e909d45-istiod-ca-cert\") pod \"e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:47.844515 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.844390 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gc69\" (UniqueName: \"kubernetes.io/projected/09df4297-042d-4762-b66a-3dab2e909d45-kube-api-access-6gc69\") pod \"e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:47.844806 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.844539 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09df4297-042d-4762-b66a-3dab2e909d45-istio-podinfo\") pod \"e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:47.844806 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.844585 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09df4297-042d-4762-b66a-3dab2e909d45-workload-socket\") pod \"e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:47.844806 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.844699 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09df4297-042d-4762-b66a-3dab2e909d45-credential-socket\") pod \"e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:47.844947 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.844820 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09df4297-042d-4762-b66a-3dab2e909d45-workload-certs\") pod \"e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:47.844947 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.844874 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09df4297-042d-4762-b66a-3dab2e909d45-workload-socket\") pod \"e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:47.845021 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.844947 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09df4297-042d-4762-b66a-3dab2e909d45-istio-data\") pod \"e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:47.845169 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.845140 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09df4297-042d-4762-b66a-3dab2e909d45-istiod-ca-cert\") pod \"e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:47.846626 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.846603 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09df4297-042d-4762-b66a-3dab2e909d45-istio-envoy\") pod \"e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:47.846947 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.846927 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09df4297-042d-4762-b66a-3dab2e909d45-istio-podinfo\") pod \"e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:47.853424 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.853386 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gc69\" (UniqueName: \"kubernetes.io/projected/09df4297-042d-4762-b66a-3dab2e909d45-kube-api-access-6gc69\") pod \"e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:47.853548 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.853513 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09df4297-042d-4762-b66a-3dab2e909d45-istio-token\") pod \"e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:47.967162 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:47.967089 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:48.106435 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:48.106388 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg"] Jul 07 20:35:48.108342 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:35:48.108311 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09df4297_042d_4762_b66a_3dab2e909d45.slice/crio-c45d61a0af88787d0e67992bd8dbc01e3911d68369520ddeff73ea1fc918cddd WatchSource:0}: Error finding container c45d61a0af88787d0e67992bd8dbc01e3911d68369520ddeff73ea1fc918cddd: Status 404 returned error can't find the container with id c45d61a0af88787d0e67992bd8dbc01e3911d68369520ddeff73ea1fc918cddd Jul 07 20:35:48.110539 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:48.110504 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 20:35:48.110648 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:48.110571 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 20:35:48.110648 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:48.110602 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 20:35:48.514667 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:48.514631 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" event={"ID":"09df4297-042d-4762-b66a-3dab2e909d45","Type":"ContainerStarted","Data":"a86ce99583e7a6042ecc2d463604c7d65dbeb16ecf5d4d45cdbd9a5325909d75"} Jul 07 20:35:48.514821 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:48.514691 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" event={"ID":"09df4297-042d-4762-b66a-3dab2e909d45","Type":"ContainerStarted","Data":"c45d61a0af88787d0e67992bd8dbc01e3911d68369520ddeff73ea1fc918cddd"} Jul 07 20:35:48.535599 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:48.535552 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" podStartSLOduration=1.535536179 podStartE2EDuration="1.535536179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:35:48.534647043 +0000 UTC m=+2068.689108994" watchObservedRunningTime="2026-07-07 20:35:48.535536179 +0000 UTC m=+2068.689998128" Jul 07 20:35:48.967795 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:48.967754 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:49.972850 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:49.972816 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:50.525357 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:50.525327 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:50.526315 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:50.526293 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:35:50.587129 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:50.587100 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h"] Jul 07 20:35:50.590242 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:50.590219 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h" Jul 07 20:35:50.592529 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:50.592508 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-e9f8cf-serving-cert\"" Jul 07 20:35:50.600747 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:50.600715 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h"] Jul 07 20:35:50.669501 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:50.669451 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/935beb3c-4e53-4a6a-91d3-19acb92f95ff-maas-api-tls\") pod \"maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h\" (UID: \"935beb3c-4e53-4a6a-91d3-19acb92f95ff\") " pod="opendatahub/maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h" Jul 07 20:35:50.669501 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:50.669503 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm4rr\" (UniqueName: \"kubernetes.io/projected/935beb3c-4e53-4a6a-91d3-19acb92f95ff-kube-api-access-vm4rr\") pod \"maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h\" (UID: \"935beb3c-4e53-4a6a-91d3-19acb92f95ff\") " pod="opendatahub/maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h" Jul 07 20:35:50.669740 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:50.669554 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/935beb3c-4e53-4a6a-91d3-19acb92f95ff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h\" (UID: \"935beb3c-4e53-4a6a-91d3-19acb92f95ff\") " pod="opendatahub/maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h" Jul 07 20:35:50.770603 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:50.770567 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/935beb3c-4e53-4a6a-91d3-19acb92f95ff-maas-api-tls\") pod \"maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h\" (UID: \"935beb3c-4e53-4a6a-91d3-19acb92f95ff\") " pod="opendatahub/maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h" Jul 07 20:35:50.770778 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:50.770616 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm4rr\" (UniqueName: \"kubernetes.io/projected/935beb3c-4e53-4a6a-91d3-19acb92f95ff-kube-api-access-vm4rr\") pod \"maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h\" (UID: \"935beb3c-4e53-4a6a-91d3-19acb92f95ff\") " pod="opendatahub/maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h" Jul 07 20:35:50.770778 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:50.770745 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/935beb3c-4e53-4a6a-91d3-19acb92f95ff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h\" (UID: \"935beb3c-4e53-4a6a-91d3-19acb92f95ff\") " pod="opendatahub/maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h" Jul 07 20:35:50.771587 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:50.771561 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/935beb3c-4e53-4a6a-91d3-19acb92f95ff-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h\" (UID: \"935beb3c-4e53-4a6a-91d3-19acb92f95ff\") " pod="opendatahub/maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h" Jul 07 20:35:50.772979 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:50.772952 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/935beb3c-4e53-4a6a-91d3-19acb92f95ff-maas-api-tls\") pod \"maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h\" (UID: \"935beb3c-4e53-4a6a-91d3-19acb92f95ff\") " pod="opendatahub/maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h" Jul 07 20:35:50.788928 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:50.788864 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm4rr\" (UniqueName: \"kubernetes.io/projected/935beb3c-4e53-4a6a-91d3-19acb92f95ff-kube-api-access-vm4rr\") pod \"maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h\" (UID: \"935beb3c-4e53-4a6a-91d3-19acb92f95ff\") " pod="opendatahub/maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h" Jul 07 20:35:50.904907 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:50.904866 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h" Jul 07 20:35:51.039496 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:51.039471 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h"] Jul 07 20:35:51.041876 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:35:51.041846 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod935beb3c_4e53_4a6a_91d3_19acb92f95ff.slice/crio-1a6da8db2389b8d31ff401f8726f492c6bfb121269800c79f596a2b502ee6e4b WatchSource:0}: Error finding container 1a6da8db2389b8d31ff401f8726f492c6bfb121269800c79f596a2b502ee6e4b: Status 404 returned error can't find the container with id 1a6da8db2389b8d31ff401f8726f492c6bfb121269800c79f596a2b502ee6e4b Jul 07 20:35:51.531905 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:51.531871 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h" event={"ID":"935beb3c-4e53-4a6a-91d3-19acb92f95ff","Type":"ContainerStarted","Data":"a41d18d85a4cca4685c5135d5adc65dcebe757c1ac9355d63a50a4660b8f8070"} Jul 07 20:35:51.532102 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:51.531957 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h" event={"ID":"935beb3c-4e53-4a6a-91d3-19acb92f95ff","Type":"ContainerStarted","Data":"1a6da8db2389b8d31ff401f8726f492c6bfb121269800c79f596a2b502ee6e4b"} Jul 07 20:35:51.551743 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:51.551693 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h" podStartSLOduration=1.201730342 podStartE2EDuration="1.551677997s" totalImagesPullingTime="349.947655ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:35:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:35:51.550275055 +0000 UTC m=+2071.704737000" watchObservedRunningTime="2026-07-07 20:35:51.551677997 +0000 UTC m=+2071.706139946" Jul 07 20:35:52.537331 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:52.537301 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h" Jul 07 20:35:58.526853 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:58.526813 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb"] Jul 07 20:35:58.531266 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:58.531241 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb" Jul 07 20:35:58.534921 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:58.534900 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-b1597e-serving-cert\"" Jul 07 20:35:58.543146 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:58.543117 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb"] Jul 07 20:35:58.548872 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:58.548840 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h" Jul 07 20:35:58.645109 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:58.645069 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvdmh\" (UniqueName: \"kubernetes.io/projected/9983b52f-3826-4ed1-906f-1140af31d129-kube-api-access-dvdmh\") pod \"maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb\" (UID: \"9983b52f-3826-4ed1-906f-1140af31d129\") " pod="opendatahub/maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb" Jul 07 20:35:58.645315 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:58.645129 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9983b52f-3826-4ed1-906f-1140af31d129-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb\" (UID: \"9983b52f-3826-4ed1-906f-1140af31d129\") " pod="opendatahub/maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb" Jul 07 20:35:58.645315 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:58.645165 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9983b52f-3826-4ed1-906f-1140af31d129-maas-api-tls\") pod \"maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb\" (UID: \"9983b52f-3826-4ed1-906f-1140af31d129\") " pod="opendatahub/maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb" Jul 07 20:35:58.746480 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:58.746437 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvdmh\" (UniqueName: \"kubernetes.io/projected/9983b52f-3826-4ed1-906f-1140af31d129-kube-api-access-dvdmh\") pod \"maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb\" (UID: \"9983b52f-3826-4ed1-906f-1140af31d129\") " pod="opendatahub/maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb" Jul 07 20:35:58.746739 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:58.746717 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9983b52f-3826-4ed1-906f-1140af31d129-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb\" (UID: \"9983b52f-3826-4ed1-906f-1140af31d129\") " pod="opendatahub/maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb" Jul 07 20:35:58.746824 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:58.746772 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9983b52f-3826-4ed1-906f-1140af31d129-maas-api-tls\") pod \"maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb\" (UID: \"9983b52f-3826-4ed1-906f-1140af31d129\") " pod="opendatahub/maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb" Jul 07 20:35:58.746880 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:35:58.746867 2565 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-b-b1597e-serving-cert: secret "maas-api-e2e-shared-b-b1597e-serving-cert" not found Jul 07 20:35:58.746939 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:35:58.746928 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9983b52f-3826-4ed1-906f-1140af31d129-maas-api-tls podName:9983b52f-3826-4ed1-906f-1140af31d129 nodeName:}" failed. No retries permitted until 2026-07-07 20:35:59.246912705 +0000 UTC m=+2079.401374633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9983b52f-3826-4ed1-906f-1140af31d129-maas-api-tls") pod "maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb" (UID: "9983b52f-3826-4ed1-906f-1140af31d129") : secret "maas-api-e2e-shared-b-b1597e-serving-cert" not found Jul 07 20:35:58.747357 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:58.747334 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9983b52f-3826-4ed1-906f-1140af31d129-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb\" (UID: \"9983b52f-3826-4ed1-906f-1140af31d129\") " pod="opendatahub/maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb" Jul 07 20:35:58.758446 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:58.758412 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvdmh\" (UniqueName: \"kubernetes.io/projected/9983b52f-3826-4ed1-906f-1140af31d129-kube-api-access-dvdmh\") pod \"maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb\" (UID: \"9983b52f-3826-4ed1-906f-1140af31d129\") " pod="opendatahub/maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb" Jul 07 20:35:59.252843 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:59.252795 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9983b52f-3826-4ed1-906f-1140af31d129-maas-api-tls\") pod \"maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb\" (UID: \"9983b52f-3826-4ed1-906f-1140af31d129\") " pod="opendatahub/maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb" Jul 07 20:35:59.255240 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:59.255222 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9983b52f-3826-4ed1-906f-1140af31d129-maas-api-tls\") pod \"maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb\" (UID: \"9983b52f-3826-4ed1-906f-1140af31d129\") " pod="opendatahub/maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb" Jul 07 20:35:59.444573 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:59.444540 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb" Jul 07 20:35:59.634783 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:35:59.634759 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb"] Jul 07 20:36:00.608900 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:00.608863 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb" event={"ID":"9983b52f-3826-4ed1-906f-1140af31d129","Type":"ContainerStarted","Data":"73547ae9d41a6bfe959bfc350db927bfbcd54e23e135d817d706260f5c4a4abd"} Jul 07 20:36:01.628225 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:01.628129 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb" event={"ID":"9983b52f-3826-4ed1-906f-1140af31d129","Type":"ContainerStarted","Data":"4219c7ad1cdde3d7937978da3dd6dbee0b8d4c86f600fe2caa2f913f6b639fdd"} Jul 07 20:36:01.628701 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:01.628254 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb" Jul 07 20:36:01.649243 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:01.649186 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb" podStartSLOduration=2.060644958 podStartE2EDuration="3.649170732s" totalImagesPullingTime="1.588525774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:36:01.646929653 +0000 UTC m=+2081.801391627" watchObservedRunningTime="2026-07-07 20:36:01.649170732 +0000 UTC m=+2081.803632681" Jul 07 20:36:04.646940 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:04.646905 2565 generic.go:356] "Generic (PLEG): container finished" podID="09df4297-042d-4762-b66a-3dab2e909d45" containerID="a86ce99583e7a6042ecc2d463604c7d65dbeb16ecf5d4d45cdbd9a5325909d75" exitCode=0 Jul 07 20:36:04.647363 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:04.646977 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" event={"ID":"09df4297-042d-4762-b66a-3dab2e909d45","Type":"ContainerDied","Data":"a86ce99583e7a6042ecc2d463604c7d65dbeb16ecf5d4d45cdbd9a5325909d75"} Jul 07 20:36:04.647458 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:04.647444 2565 scope.go:117] "RemoveContainer" containerID="a86ce99583e7a6042ecc2d463604c7d65dbeb16ecf5d4d45cdbd9a5325909d75" Jul 07 20:36:04.648440 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:04.648395 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 20:36:04.648554 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:04.648479 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 20:36:04.648554 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:04.648520 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 20:36:05.525805 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:05.525768 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:36:05.655796 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:05.655766 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" event={"ID":"09df4297-042d-4762-b66a-3dab2e909d45","Type":"ContainerStarted","Data":"a66f5f0fc9b36b2b565a13e5750bcbfbf554d5ade91b7133986bc5676ff31911"} Jul 07 20:36:05.969228 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:05.969190 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:36:05.969469 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:05.969343 2565 patch_prober.go:28] interesting pod/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 07 20:36:05.969469 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:05.969392 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" podUID="09df4297-042d-4762-b66a-3dab2e909d45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 07 20:36:06.968008 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:06.967965 2565 patch_prober.go:28] interesting pod/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 07 20:36:06.968537 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:06.968040 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" podUID="09df4297-042d-4762-b66a-3dab2e909d45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 07 20:36:07.645772 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:07.645746 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb" Jul 07 20:36:07.968700 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:07.968585 2565 patch_prober.go:28] interesting pod/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 07 20:36:07.969312 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:07.969275 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" podUID="09df4297-042d-4762-b66a-3dab2e909d45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 07 20:36:08.968207 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:08.968165 2565 patch_prober.go:28] interesting pod/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 07 20:36:08.968447 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:08.968232 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" podUID="09df4297-042d-4762-b66a-3dab2e909d45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 07 20:36:09.971879 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:09.971849 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:36:09.972266 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:09.972123 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:36:09.972835 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:09.972818 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:36:57.178256 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:57.178224 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7"] Jul 07 20:36:57.181567 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:57.181540 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" Jul 07 20:36:57.188393 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:57.188366 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b1597e\"/\"kube-root-ca.crt\"" Jul 07 20:36:57.188547 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:57.188366 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b1597e\"/\"default-dockercfg-8stff\"" Jul 07 20:36:57.188547 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:57.188427 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b1597e\"/\"openshift-service-ca.crt\"" Jul 07 20:36:57.188547 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:57.188372 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b1597e\"/\"auth-test-model-b1597e-kserve-self-signed-certs\"" Jul 07 20:36:57.194497 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:57.194476 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7"] Jul 07 20:36:57.303717 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:57.303680 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zvvs\" (UniqueName: \"kubernetes.io/projected/793ed65b-abcd-47c6-9d42-387053edf827-kube-api-access-2zvvs\") pod \"auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7\" (UID: \"793ed65b-abcd-47c6-9d42-387053edf827\") " pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" Jul 07 20:36:57.303918 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:57.303737 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/793ed65b-abcd-47c6-9d42-387053edf827-model-cache\") pod \"auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7\" (UID: \"793ed65b-abcd-47c6-9d42-387053edf827\") " pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" Jul 07 20:36:57.303918 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:57.303775 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/793ed65b-abcd-47c6-9d42-387053edf827-tmp-dir\") pod \"auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7\" (UID: \"793ed65b-abcd-47c6-9d42-387053edf827\") " pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" Jul 07 20:36:57.303918 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:57.303840 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/793ed65b-abcd-47c6-9d42-387053edf827-dshm\") pod \"auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7\" (UID: \"793ed65b-abcd-47c6-9d42-387053edf827\") " pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" Jul 07 20:36:57.303918 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:57.303868 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/793ed65b-abcd-47c6-9d42-387053edf827-home\") pod \"auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7\" (UID: \"793ed65b-abcd-47c6-9d42-387053edf827\") " pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" Jul 07 20:36:57.304093 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:57.303929 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/793ed65b-abcd-47c6-9d42-387053edf827-tls-certs\") pod \"auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7\" (UID: \"793ed65b-abcd-47c6-9d42-387053edf827\") " pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" Jul 07 20:36:57.404681 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:57.404632 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/793ed65b-abcd-47c6-9d42-387053edf827-dshm\") pod \"auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7\" (UID: \"793ed65b-abcd-47c6-9d42-387053edf827\") " pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" Jul 07 20:36:57.404895 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:57.404690 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/793ed65b-abcd-47c6-9d42-387053edf827-home\") pod \"auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7\" (UID: \"793ed65b-abcd-47c6-9d42-387053edf827\") " pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" Jul 07 20:36:57.404895 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:57.404751 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/793ed65b-abcd-47c6-9d42-387053edf827-tls-certs\") pod \"auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7\" (UID: \"793ed65b-abcd-47c6-9d42-387053edf827\") " pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" Jul 07 20:36:57.404895 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:57.404821 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zvvs\" (UniqueName: \"kubernetes.io/projected/793ed65b-abcd-47c6-9d42-387053edf827-kube-api-access-2zvvs\") pod \"auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7\" (UID: \"793ed65b-abcd-47c6-9d42-387053edf827\") " pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" Jul 07 20:36:57.404895 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:57.404861 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/793ed65b-abcd-47c6-9d42-387053edf827-model-cache\") pod \"auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7\" (UID: \"793ed65b-abcd-47c6-9d42-387053edf827\") " pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" Jul 07 20:36:57.404895 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:57.404887 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/793ed65b-abcd-47c6-9d42-387053edf827-tmp-dir\") pod \"auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7\" (UID: \"793ed65b-abcd-47c6-9d42-387053edf827\") " pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" Jul 07 20:36:57.405189 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:57.405145 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/793ed65b-abcd-47c6-9d42-387053edf827-home\") pod \"auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7\" (UID: \"793ed65b-abcd-47c6-9d42-387053edf827\") " pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" Jul 07 20:36:57.405189 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:57.405182 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/793ed65b-abcd-47c6-9d42-387053edf827-model-cache\") pod \"auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7\" (UID: \"793ed65b-abcd-47c6-9d42-387053edf827\") " pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" Jul 07 20:36:57.405330 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:57.405299 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/793ed65b-abcd-47c6-9d42-387053edf827-tmp-dir\") pod \"auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7\" (UID: \"793ed65b-abcd-47c6-9d42-387053edf827\") " pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" Jul 07 20:36:57.406885 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:57.406862 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/793ed65b-abcd-47c6-9d42-387053edf827-dshm\") pod \"auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7\" (UID: \"793ed65b-abcd-47c6-9d42-387053edf827\") " pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" Jul 07 20:36:57.407142 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:57.407126 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/793ed65b-abcd-47c6-9d42-387053edf827-tls-certs\") pod \"auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7\" (UID: \"793ed65b-abcd-47c6-9d42-387053edf827\") " pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" Jul 07 20:36:57.415680 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:57.415651 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zvvs\" (UniqueName: \"kubernetes.io/projected/793ed65b-abcd-47c6-9d42-387053edf827-kube-api-access-2zvvs\") pod \"auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7\" (UID: \"793ed65b-abcd-47c6-9d42-387053edf827\") " pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" Jul 07 20:36:57.495588 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:57.495513 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" Jul 07 20:36:57.627795 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:57.627765 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7"] Jul 07 20:36:57.629821 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:36:57.629796 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod793ed65b_abcd_47c6_9d42_387053edf827.slice/crio-71491ff1c6884ed68dbf1b7ac027ed5682237080c63bc2eaaf22882ebd01f96a WatchSource:0}: Error finding container 71491ff1c6884ed68dbf1b7ac027ed5682237080c63bc2eaaf22882ebd01f96a: Status 404 returned error can't find the container with id 71491ff1c6884ed68dbf1b7ac027ed5682237080c63bc2eaaf22882ebd01f96a Jul 07 20:36:57.915368 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:57.915334 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" event={"ID":"793ed65b-abcd-47c6-9d42-387053edf827","Type":"ContainerStarted","Data":"4b747720093fc9c7188d2847d08d71d21637520afb05f01ad6f7b983f2760153"} Jul 07 20:36:57.915543 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:57.915380 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" event={"ID":"793ed65b-abcd-47c6-9d42-387053edf827","Type":"ContainerStarted","Data":"71491ff1c6884ed68dbf1b7ac027ed5682237080c63bc2eaaf22882ebd01f96a"} Jul 07 20:36:57.934553 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:36:57.934506 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" podStartSLOduration=0.93449164 podStartE2EDuration="934.49164ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:36:57.933200537 +0000 UTC m=+2138.087662486" watchObservedRunningTime="2026-07-07 20:36:57.93449164 +0000 UTC m=+2138.088953587" Jul 07 20:37:07.496467 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:37:07.496422 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" Jul 07 20:37:07.496467 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:37:07.496479 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" Jul 07 20:37:07.508886 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:37:07.508857 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" Jul 07 20:37:07.976501 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:37:07.976471 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" Jul 07 20:38:58.760048 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:38:58.759960 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7"] Jul 07 20:39:13.780510 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:13.780445 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" podUID="793ed65b-abcd-47c6-9d42-387053edf827" containerName="main" containerID="cri-o://4b747720093fc9c7188d2847d08d71d21637520afb05f01ad6f7b983f2760153" gracePeriod=45 Jul 07 20:39:14.028723 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:14.028695 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" Jul 07 20:39:14.135676 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:14.135645 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/793ed65b-abcd-47c6-9d42-387053edf827-dshm\") pod \"793ed65b-abcd-47c6-9d42-387053edf827\" (UID: \"793ed65b-abcd-47c6-9d42-387053edf827\") " Jul 07 20:39:14.135863 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:14.135683 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/793ed65b-abcd-47c6-9d42-387053edf827-tls-certs\") pod \"793ed65b-abcd-47c6-9d42-387053edf827\" (UID: \"793ed65b-abcd-47c6-9d42-387053edf827\") " Jul 07 20:39:14.135863 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:14.135763 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/793ed65b-abcd-47c6-9d42-387053edf827-tmp-dir\") pod \"793ed65b-abcd-47c6-9d42-387053edf827\" (UID: \"793ed65b-abcd-47c6-9d42-387053edf827\") " Jul 07 20:39:14.135863 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:14.135830 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zvvs\" (UniqueName: \"kubernetes.io/projected/793ed65b-abcd-47c6-9d42-387053edf827-kube-api-access-2zvvs\") pod \"793ed65b-abcd-47c6-9d42-387053edf827\" (UID: \"793ed65b-abcd-47c6-9d42-387053edf827\") " Jul 07 20:39:14.136024 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:14.135888 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/793ed65b-abcd-47c6-9d42-387053edf827-model-cache\") pod \"793ed65b-abcd-47c6-9d42-387053edf827\" (UID: \"793ed65b-abcd-47c6-9d42-387053edf827\") " Jul 07 20:39:14.136024 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:14.135927 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/793ed65b-abcd-47c6-9d42-387053edf827-home\") pod \"793ed65b-abcd-47c6-9d42-387053edf827\" (UID: \"793ed65b-abcd-47c6-9d42-387053edf827\") " Jul 07 20:39:14.136144 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:14.136109 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793ed65b-abcd-47c6-9d42-387053edf827-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "793ed65b-abcd-47c6-9d42-387053edf827" (UID: "793ed65b-abcd-47c6-9d42-387053edf827"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:39:14.136144 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:14.136125 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793ed65b-abcd-47c6-9d42-387053edf827-model-cache" (OuterVolumeSpecName: "model-cache") pod "793ed65b-abcd-47c6-9d42-387053edf827" (UID: "793ed65b-abcd-47c6-9d42-387053edf827"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:39:14.136250 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:14.136217 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793ed65b-abcd-47c6-9d42-387053edf827-home" (OuterVolumeSpecName: "home") pod "793ed65b-abcd-47c6-9d42-387053edf827" (UID: "793ed65b-abcd-47c6-9d42-387053edf827"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:39:14.136250 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:14.136241 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/793ed65b-abcd-47c6-9d42-387053edf827-model-cache\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:39:14.136334 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:14.136260 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/793ed65b-abcd-47c6-9d42-387053edf827-tmp-dir\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:39:14.137962 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:14.137940 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793ed65b-abcd-47c6-9d42-387053edf827-dshm" (OuterVolumeSpecName: "dshm") pod "793ed65b-abcd-47c6-9d42-387053edf827" (UID: "793ed65b-abcd-47c6-9d42-387053edf827"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:39:14.138064 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:14.137987 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/793ed65b-abcd-47c6-9d42-387053edf827-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "793ed65b-abcd-47c6-9d42-387053edf827" (UID: "793ed65b-abcd-47c6-9d42-387053edf827"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:39:14.138064 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:14.138018 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/793ed65b-abcd-47c6-9d42-387053edf827-kube-api-access-2zvvs" (OuterVolumeSpecName: "kube-api-access-2zvvs") pod "793ed65b-abcd-47c6-9d42-387053edf827" (UID: "793ed65b-abcd-47c6-9d42-387053edf827"). InnerVolumeSpecName "kube-api-access-2zvvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 20:39:14.237619 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:14.237583 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/793ed65b-abcd-47c6-9d42-387053edf827-home\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:39:14.237619 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:14.237611 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/793ed65b-abcd-47c6-9d42-387053edf827-dshm\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:39:14.237619 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:14.237620 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/793ed65b-abcd-47c6-9d42-387053edf827-tls-certs\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:39:14.237619 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:14.237630 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zvvs\" (UniqueName: \"kubernetes.io/projected/793ed65b-abcd-47c6-9d42-387053edf827-kube-api-access-2zvvs\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:39:14.610931 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:14.610844 2565 generic.go:356] "Generic (PLEG): container finished" podID="793ed65b-abcd-47c6-9d42-387053edf827" containerID="4b747720093fc9c7188d2847d08d71d21637520afb05f01ad6f7b983f2760153" exitCode=0 Jul 07 20:39:14.610931 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:14.610896 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" Jul 07 20:39:14.610931 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:14.610918 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" event={"ID":"793ed65b-abcd-47c6-9d42-387053edf827","Type":"ContainerDied","Data":"4b747720093fc9c7188d2847d08d71d21637520afb05f01ad6f7b983f2760153"} Jul 07 20:39:14.611142 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:14.610949 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7" event={"ID":"793ed65b-abcd-47c6-9d42-387053edf827","Type":"ContainerDied","Data":"71491ff1c6884ed68dbf1b7ac027ed5682237080c63bc2eaaf22882ebd01f96a"} Jul 07 20:39:14.611142 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:14.610967 2565 scope.go:117] "RemoveContainer" containerID="4b747720093fc9c7188d2847d08d71d21637520afb05f01ad6f7b983f2760153" Jul 07 20:39:14.619972 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:14.619955 2565 scope.go:117] "RemoveContainer" containerID="4b747720093fc9c7188d2847d08d71d21637520afb05f01ad6f7b983f2760153" Jul 07 20:39:14.620221 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:39:14.620204 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b747720093fc9c7188d2847d08d71d21637520afb05f01ad6f7b983f2760153\": container with ID starting with 4b747720093fc9c7188d2847d08d71d21637520afb05f01ad6f7b983f2760153 not found: ID does not exist" containerID="4b747720093fc9c7188d2847d08d71d21637520afb05f01ad6f7b983f2760153" Jul 07 20:39:14.620273 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:14.620227 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b747720093fc9c7188d2847d08d71d21637520afb05f01ad6f7b983f2760153"} err="failed to get container status \"4b747720093fc9c7188d2847d08d71d21637520afb05f01ad6f7b983f2760153\": rpc error: code = NotFound desc = could not find container \"4b747720093fc9c7188d2847d08d71d21637520afb05f01ad6f7b983f2760153\": container with ID starting with 4b747720093fc9c7188d2847d08d71d21637520afb05f01ad6f7b983f2760153 not found: ID does not exist" Jul 07 20:39:14.635860 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:14.635840 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7"] Jul 07 20:39:14.639666 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:14.639645 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b1597e/auth-test-model-b1597e-kserve-7fd6d76bf9-mqst7"] Jul 07 20:39:15.684873 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:15.684840 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="793ed65b-abcd-47c6-9d42-387053edf827" path="/var/lib/kubelet/pods/793ed65b-abcd-47c6-9d42-387053edf827/volumes" Jul 07 20:39:50.571295 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:50.571259 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll"] Jul 07 20:39:50.571773 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:50.571674 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="793ed65b-abcd-47c6-9d42-387053edf827" containerName="main" Jul 07 20:39:50.571773 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:50.571687 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="793ed65b-abcd-47c6-9d42-387053edf827" containerName="main" Jul 07 20:39:50.571773 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:50.571757 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="793ed65b-abcd-47c6-9d42-387053edf827" containerName="main" Jul 07 20:39:50.574920 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:50.574899 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" Jul 07 20:39:50.578797 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:50.578770 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b1597e\"/\"openshift-service-ca.crt\"" Jul 07 20:39:50.578931 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:50.578815 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b1597e\"/\"kube-root-ca.crt\"" Jul 07 20:39:50.578931 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:50.578828 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b1597e\"/\"sub-test-model-b1597e-kserve-self-signed-certs\"" Jul 07 20:39:50.579051 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:50.578980 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b1597e\"/\"default-dockercfg-8stff\"" Jul 07 20:39:50.585872 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:50.585839 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll"] Jul 07 20:39:50.676725 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:50.676672 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk8r7\" (UniqueName: \"kubernetes.io/projected/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-kube-api-access-qk8r7\") pod \"sub-test-model-b1597e-kserve-54b877bc97-j9cll\" (UID: \"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac\") " pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" Jul 07 20:39:50.676883 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:50.676786 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-model-cache\") pod \"sub-test-model-b1597e-kserve-54b877bc97-j9cll\" (UID: \"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac\") " pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" Jul 07 20:39:50.676883 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:50.676823 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-dshm\") pod \"sub-test-model-b1597e-kserve-54b877bc97-j9cll\" (UID: \"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac\") " pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" Jul 07 20:39:50.676962 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:50.676916 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-tls-certs\") pod \"sub-test-model-b1597e-kserve-54b877bc97-j9cll\" (UID: \"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac\") " pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" Jul 07 20:39:50.676998 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:50.676960 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-home\") pod \"sub-test-model-b1597e-kserve-54b877bc97-j9cll\" (UID: \"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac\") " pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" Jul 07 20:39:50.676998 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:50.676975 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-tmp-dir\") pod \"sub-test-model-b1597e-kserve-54b877bc97-j9cll\" (UID: \"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac\") " pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" Jul 07 20:39:50.777705 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:50.777673 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-model-cache\") pod \"sub-test-model-b1597e-kserve-54b877bc97-j9cll\" (UID: \"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac\") " pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" Jul 07 20:39:50.777870 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:50.777713 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-dshm\") pod \"sub-test-model-b1597e-kserve-54b877bc97-j9cll\" (UID: \"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac\") " pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" Jul 07 20:39:50.777870 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:50.777780 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-tls-certs\") pod \"sub-test-model-b1597e-kserve-54b877bc97-j9cll\" (UID: \"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac\") " pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" Jul 07 20:39:50.777870 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:50.777803 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-home\") pod \"sub-test-model-b1597e-kserve-54b877bc97-j9cll\" (UID: \"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac\") " pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" Jul 07 20:39:50.777870 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:50.777821 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-tmp-dir\") pod \"sub-test-model-b1597e-kserve-54b877bc97-j9cll\" (UID: \"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac\") " pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" Jul 07 20:39:50.777870 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:50.777847 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk8r7\" (UniqueName: \"kubernetes.io/projected/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-kube-api-access-qk8r7\") pod \"sub-test-model-b1597e-kserve-54b877bc97-j9cll\" (UID: \"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac\") " pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" Jul 07 20:39:50.778172 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:50.778147 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-model-cache\") pod \"sub-test-model-b1597e-kserve-54b877bc97-j9cll\" (UID: \"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac\") " pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" Jul 07 20:39:50.778568 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:50.778202 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-home\") pod \"sub-test-model-b1597e-kserve-54b877bc97-j9cll\" (UID: \"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac\") " pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" Jul 07 20:39:50.785416 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:50.778979 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-tmp-dir\") pod \"sub-test-model-b1597e-kserve-54b877bc97-j9cll\" (UID: \"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac\") " pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" Jul 07 20:39:50.785416 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:50.780299 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-dshm\") pod \"sub-test-model-b1597e-kserve-54b877bc97-j9cll\" (UID: \"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac\") " pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" Jul 07 20:39:50.788330 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:50.788302 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk8r7\" (UniqueName: \"kubernetes.io/projected/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-kube-api-access-qk8r7\") pod \"sub-test-model-b1597e-kserve-54b877bc97-j9cll\" (UID: \"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac\") " pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" Jul 07 20:39:50.789812 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:50.789792 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-tls-certs\") pod \"sub-test-model-b1597e-kserve-54b877bc97-j9cll\" (UID: \"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac\") " pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" Jul 07 20:39:50.885297 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:50.885260 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" Jul 07 20:39:51.019105 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:51.019080 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll"] Jul 07 20:39:51.021234 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:39:51.021206 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a992eb9_ea2c_4f30_bbdd_f7bd9689b2ac.slice/crio-ca1c732cfa6dd2122c43ae12ac6bb534242f4835c648e85cd215aab40ae668fc WatchSource:0}: Error finding container ca1c732cfa6dd2122c43ae12ac6bb534242f4835c648e85cd215aab40ae668fc: Status 404 returned error can't find the container with id ca1c732cfa6dd2122c43ae12ac6bb534242f4835c648e85cd215aab40ae668fc Jul 07 20:39:51.799328 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:51.799290 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" event={"ID":"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac","Type":"ContainerStarted","Data":"b1f882c390a32edf4aea0ca4f2857404c3ee4ebb156ccbcbe78f59e1693b74e6"} Jul 07 20:39:51.799727 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:51.799336 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" event={"ID":"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac","Type":"ContainerStarted","Data":"ca1c732cfa6dd2122c43ae12ac6bb534242f4835c648e85cd215aab40ae668fc"} Jul 07 20:39:51.822572 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:39:51.822515 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" podStartSLOduration=1.822497797 podStartE2EDuration="1.822497797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:39:51.820486743 +0000 UTC m=+2311.974948691" watchObservedRunningTime="2026-07-07 20:39:51.822497797 +0000 UTC m=+2311.976959744" Jul 07 20:40:00.885637 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:40:00.885595 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" Jul 07 20:40:00.886242 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:40:00.885735 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" Jul 07 20:40:00.897891 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:40:00.897871 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" Jul 07 20:40:01.862807 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:40:01.862779 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" Jul 07 20:40:48.075971 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:40:48.075927 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll"] Jul 07 20:41:03.096999 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:03.096955 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" podUID="1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac" containerName="main" containerID="cri-o://b1f882c390a32edf4aea0ca4f2857404c3ee4ebb156ccbcbe78f59e1693b74e6" gracePeriod=45 Jul 07 20:41:03.352906 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:03.352849 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" Jul 07 20:41:03.391233 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:03.391202 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-tmp-dir\") pod \"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac\" (UID: \"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac\") " Jul 07 20:41:03.391446 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:03.391268 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-tls-certs\") pod \"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac\" (UID: \"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac\") " Jul 07 20:41:03.391446 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:03.391299 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qk8r7\" (UniqueName: \"kubernetes.io/projected/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-kube-api-access-qk8r7\") pod \"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac\" (UID: \"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac\") " Jul 07 20:41:03.391446 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:03.391314 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-home\") pod \"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac\" (UID: \"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac\") " Jul 07 20:41:03.391446 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:03.391344 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-dshm\") pod \"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac\" (UID: \"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac\") " Jul 07 20:41:03.391446 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:03.391419 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-model-cache\") pod \"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac\" (UID: \"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac\") " Jul 07 20:41:03.391725 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:03.391532 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac" (UID: "1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:41:03.391725 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:03.391617 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-home" (OuterVolumeSpecName: "home") pod "1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac" (UID: "1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:41:03.391830 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:03.391729 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac" (UID: "1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:41:03.391883 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:03.391857 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-home\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:41:03.391883 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:03.391871 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-model-cache\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:41:03.391974 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:03.391885 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-tmp-dir\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:41:03.393441 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:03.393414 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac" (UID: "1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:41:03.393866 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:03.393842 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-dshm" (OuterVolumeSpecName: "dshm") pod "1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac" (UID: "1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:41:03.393952 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:03.393862 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-kube-api-access-qk8r7" (OuterVolumeSpecName: "kube-api-access-qk8r7") pod "1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac" (UID: "1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac"). InnerVolumeSpecName "kube-api-access-qk8r7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 20:41:03.492555 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:03.492517 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-tls-certs\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:41:03.492555 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:03.492550 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk8r7\" (UniqueName: \"kubernetes.io/projected/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-kube-api-access-qk8r7\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:41:03.492555 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:03.492559 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac-dshm\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:41:04.165371 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:04.165335 2565 generic.go:356] "Generic (PLEG): container finished" podID="1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac" containerID="b1f882c390a32edf4aea0ca4f2857404c3ee4ebb156ccbcbe78f59e1693b74e6" exitCode=0 Jul 07 20:41:04.167603 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:04.165381 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" Jul 07 20:41:04.167603 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:04.165431 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" event={"ID":"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac","Type":"ContainerDied","Data":"b1f882c390a32edf4aea0ca4f2857404c3ee4ebb156ccbcbe78f59e1693b74e6"} Jul 07 20:41:04.167603 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:04.165472 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll" event={"ID":"1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac","Type":"ContainerDied","Data":"ca1c732cfa6dd2122c43ae12ac6bb534242f4835c648e85cd215aab40ae668fc"} Jul 07 20:41:04.167603 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:04.165491 2565 scope.go:117] "RemoveContainer" containerID="b1f882c390a32edf4aea0ca4f2857404c3ee4ebb156ccbcbe78f59e1693b74e6" Jul 07 20:41:04.174081 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:04.174056 2565 scope.go:117] "RemoveContainer" containerID="b1f882c390a32edf4aea0ca4f2857404c3ee4ebb156ccbcbe78f59e1693b74e6" Jul 07 20:41:04.174338 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:41:04.174320 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1f882c390a32edf4aea0ca4f2857404c3ee4ebb156ccbcbe78f59e1693b74e6\": container with ID starting with b1f882c390a32edf4aea0ca4f2857404c3ee4ebb156ccbcbe78f59e1693b74e6 not found: ID does not exist" containerID="b1f882c390a32edf4aea0ca4f2857404c3ee4ebb156ccbcbe78f59e1693b74e6" Jul 07 20:41:04.174390 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:04.174350 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1f882c390a32edf4aea0ca4f2857404c3ee4ebb156ccbcbe78f59e1693b74e6"} err="failed to get container status \"b1f882c390a32edf4aea0ca4f2857404c3ee4ebb156ccbcbe78f59e1693b74e6\": rpc error: code = NotFound desc = could not find container \"b1f882c390a32edf4aea0ca4f2857404c3ee4ebb156ccbcbe78f59e1693b74e6\": container with ID starting with b1f882c390a32edf4aea0ca4f2857404c3ee4ebb156ccbcbe78f59e1693b74e6 not found: ID does not exist" Jul 07 20:41:04.182263 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:04.182243 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll"] Jul 07 20:41:04.191061 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:04.191034 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b1597e/sub-test-model-b1597e-kserve-54b877bc97-j9cll"] Jul 07 20:41:05.685060 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:05.685026 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac" path="/var/lib/kubelet/pods/1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac/volumes" Jul 07 20:41:29.476201 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:29.476166 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl"] Jul 07 20:41:29.476666 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:29.476582 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac" containerName="main" Jul 07 20:41:29.476666 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:29.476593 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac" containerName="main" Jul 07 20:41:29.476666 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:29.476663 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a992eb9-ea2c-4f30-bbdd-f7bd9689b2ac" containerName="main" Jul 07 20:41:29.479709 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:29.479693 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" Jul 07 20:41:29.483566 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:29.483544 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b1597e\"/\"default-dockercfg-8stff\"" Jul 07 20:41:29.483697 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:29.483634 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b1597e\"/\"kube-root-ca.crt\"" Jul 07 20:41:29.483697 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:29.483671 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b1597e\"/\"openshift-service-ca.crt\"" Jul 07 20:41:29.483697 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:29.483643 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b1597e\"/\"rate-test-model-b1597e-kserve-self-signed-certs\"" Jul 07 20:41:29.487529 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:29.487512 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl"] Jul 07 20:41:29.546523 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:29.546494 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c430a278-d7f9-4482-9267-98fab817bb08-home\") pod \"rate-test-model-b1597e-kserve-75bb78ff75-k5qgl\" (UID: \"c430a278-d7f9-4482-9267-98fab817bb08\") " pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" Jul 07 20:41:29.546665 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:29.546527 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c430a278-d7f9-4482-9267-98fab817bb08-dshm\") pod \"rate-test-model-b1597e-kserve-75bb78ff75-k5qgl\" (UID: \"c430a278-d7f9-4482-9267-98fab817bb08\") " pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" Jul 07 20:41:29.546665 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:29.546550 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqb8m\" (UniqueName: \"kubernetes.io/projected/c430a278-d7f9-4482-9267-98fab817bb08-kube-api-access-pqb8m\") pod \"rate-test-model-b1597e-kserve-75bb78ff75-k5qgl\" (UID: \"c430a278-d7f9-4482-9267-98fab817bb08\") " pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" Jul 07 20:41:29.546749 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:29.546670 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c430a278-d7f9-4482-9267-98fab817bb08-tmp-dir\") pod \"rate-test-model-b1597e-kserve-75bb78ff75-k5qgl\" (UID: \"c430a278-d7f9-4482-9267-98fab817bb08\") " pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" Jul 07 20:41:29.546749 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:29.546694 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c430a278-d7f9-4482-9267-98fab817bb08-tls-certs\") pod \"rate-test-model-b1597e-kserve-75bb78ff75-k5qgl\" (UID: \"c430a278-d7f9-4482-9267-98fab817bb08\") " pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" Jul 07 20:41:29.546826 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:29.546765 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c430a278-d7f9-4482-9267-98fab817bb08-model-cache\") pod \"rate-test-model-b1597e-kserve-75bb78ff75-k5qgl\" (UID: \"c430a278-d7f9-4482-9267-98fab817bb08\") " pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" Jul 07 20:41:29.647885 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:29.647854 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c430a278-d7f9-4482-9267-98fab817bb08-tmp-dir\") pod \"rate-test-model-b1597e-kserve-75bb78ff75-k5qgl\" (UID: \"c430a278-d7f9-4482-9267-98fab817bb08\") " pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" Jul 07 20:41:29.647885 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:29.647883 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c430a278-d7f9-4482-9267-98fab817bb08-tls-certs\") pod \"rate-test-model-b1597e-kserve-75bb78ff75-k5qgl\" (UID: \"c430a278-d7f9-4482-9267-98fab817bb08\") " pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" Jul 07 20:41:29.648078 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:29.647922 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c430a278-d7f9-4482-9267-98fab817bb08-model-cache\") pod \"rate-test-model-b1597e-kserve-75bb78ff75-k5qgl\" (UID: \"c430a278-d7f9-4482-9267-98fab817bb08\") " pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" Jul 07 20:41:29.648078 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:29.648036 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c430a278-d7f9-4482-9267-98fab817bb08-home\") pod \"rate-test-model-b1597e-kserve-75bb78ff75-k5qgl\" (UID: \"c430a278-d7f9-4482-9267-98fab817bb08\") " pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" Jul 07 20:41:29.648078 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:29.648067 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c430a278-d7f9-4482-9267-98fab817bb08-dshm\") pod \"rate-test-model-b1597e-kserve-75bb78ff75-k5qgl\" (UID: \"c430a278-d7f9-4482-9267-98fab817bb08\") " pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" Jul 07 20:41:29.648206 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:29.648099 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqb8m\" (UniqueName: \"kubernetes.io/projected/c430a278-d7f9-4482-9267-98fab817bb08-kube-api-access-pqb8m\") pod \"rate-test-model-b1597e-kserve-75bb78ff75-k5qgl\" (UID: \"c430a278-d7f9-4482-9267-98fab817bb08\") " pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" Jul 07 20:41:29.648293 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:29.648263 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c430a278-d7f9-4482-9267-98fab817bb08-tmp-dir\") pod \"rate-test-model-b1597e-kserve-75bb78ff75-k5qgl\" (UID: \"c430a278-d7f9-4482-9267-98fab817bb08\") " pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" Jul 07 20:41:29.648359 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:29.648331 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c430a278-d7f9-4482-9267-98fab817bb08-model-cache\") pod \"rate-test-model-b1597e-kserve-75bb78ff75-k5qgl\" (UID: \"c430a278-d7f9-4482-9267-98fab817bb08\") " pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" Jul 07 20:41:29.648422 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:29.648347 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c430a278-d7f9-4482-9267-98fab817bb08-home\") pod \"rate-test-model-b1597e-kserve-75bb78ff75-k5qgl\" (UID: \"c430a278-d7f9-4482-9267-98fab817bb08\") " pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" Jul 07 20:41:29.650253 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:29.650231 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c430a278-d7f9-4482-9267-98fab817bb08-dshm\") pod \"rate-test-model-b1597e-kserve-75bb78ff75-k5qgl\" (UID: \"c430a278-d7f9-4482-9267-98fab817bb08\") " pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" Jul 07 20:41:29.650496 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:29.650476 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c430a278-d7f9-4482-9267-98fab817bb08-tls-certs\") pod \"rate-test-model-b1597e-kserve-75bb78ff75-k5qgl\" (UID: \"c430a278-d7f9-4482-9267-98fab817bb08\") " pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" Jul 07 20:41:29.657230 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:29.657212 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqb8m\" (UniqueName: \"kubernetes.io/projected/c430a278-d7f9-4482-9267-98fab817bb08-kube-api-access-pqb8m\") pod \"rate-test-model-b1597e-kserve-75bb78ff75-k5qgl\" (UID: \"c430a278-d7f9-4482-9267-98fab817bb08\") " pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" Jul 07 20:41:29.790128 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:29.790054 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" Jul 07 20:41:29.927742 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:29.927714 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl"] Jul 07 20:41:29.929717 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:41:29.929680 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc430a278_d7f9_4482_9267_98fab817bb08.slice/crio-3768d4171777388a17c7bac2616e8e61bdbfba711c72925b6e8f8754cd7228c9 WatchSource:0}: Error finding container 3768d4171777388a17c7bac2616e8e61bdbfba711c72925b6e8f8754cd7228c9: Status 404 returned error can't find the container with id 3768d4171777388a17c7bac2616e8e61bdbfba711c72925b6e8f8754cd7228c9 Jul 07 20:41:30.293111 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:30.293074 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" event={"ID":"c430a278-d7f9-4482-9267-98fab817bb08","Type":"ContainerStarted","Data":"aeac4c5f16c357d9f5d5fa116884628128406e8d5be8994aa017f3b3fd98e158"} Jul 07 20:41:30.293299 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:30.293118 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" event={"ID":"c430a278-d7f9-4482-9267-98fab817bb08","Type":"ContainerStarted","Data":"3768d4171777388a17c7bac2616e8e61bdbfba711c72925b6e8f8754cd7228c9"} Jul 07 20:41:30.314369 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:30.314322 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" podStartSLOduration=1.314309444 podStartE2EDuration="1.314309444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:41:30.312195643 +0000 UTC m=+2410.466657593" watchObservedRunningTime="2026-07-07 20:41:30.314309444 +0000 UTC m=+2410.468771395" Jul 07 20:41:39.790459 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:39.790425 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" Jul 07 20:41:39.790955 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:39.790503 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" Jul 07 20:41:39.802689 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:39.802667 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" Jul 07 20:41:40.355383 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:41:40.355355 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" Jul 07 20:42:08.157628 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:08.157596 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl"] Jul 07 20:42:09.997923 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:09.997881 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h"] Jul 07 20:42:09.998492 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:09.998134 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h" podUID="935beb3c-4e53-4a6a-91d3-19acb92f95ff" containerName="maas-api" containerID="cri-o://a41d18d85a4cca4685c5135d5adc65dcebe757c1ac9355d63a50a4660b8f8070" gracePeriod=30 Jul 07 20:42:10.443346 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:10.443321 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h" Jul 07 20:42:10.499074 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:10.499039 2565 generic.go:356] "Generic (PLEG): container finished" podID="935beb3c-4e53-4a6a-91d3-19acb92f95ff" containerID="a41d18d85a4cca4685c5135d5adc65dcebe757c1ac9355d63a50a4660b8f8070" exitCode=0 Jul 07 20:42:10.499277 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:10.499088 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h" Jul 07 20:42:10.499277 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:10.499129 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h" event={"ID":"935beb3c-4e53-4a6a-91d3-19acb92f95ff","Type":"ContainerDied","Data":"a41d18d85a4cca4685c5135d5adc65dcebe757c1ac9355d63a50a4660b8f8070"} Jul 07 20:42:10.499277 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:10.499188 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h" event={"ID":"935beb3c-4e53-4a6a-91d3-19acb92f95ff","Type":"ContainerDied","Data":"1a6da8db2389b8d31ff401f8726f492c6bfb121269800c79f596a2b502ee6e4b"} Jul 07 20:42:10.499277 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:10.499217 2565 scope.go:117] "RemoveContainer" containerID="a41d18d85a4cca4685c5135d5adc65dcebe757c1ac9355d63a50a4660b8f8070" Jul 07 20:42:10.507748 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:10.507729 2565 scope.go:117] "RemoveContainer" containerID="a41d18d85a4cca4685c5135d5adc65dcebe757c1ac9355d63a50a4660b8f8070" Jul 07 20:42:10.508001 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:42:10.507983 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a41d18d85a4cca4685c5135d5adc65dcebe757c1ac9355d63a50a4660b8f8070\": container with ID starting with a41d18d85a4cca4685c5135d5adc65dcebe757c1ac9355d63a50a4660b8f8070 not found: ID does not exist" containerID="a41d18d85a4cca4685c5135d5adc65dcebe757c1ac9355d63a50a4660b8f8070" Jul 07 20:42:10.508046 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:10.508010 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a41d18d85a4cca4685c5135d5adc65dcebe757c1ac9355d63a50a4660b8f8070"} err="failed to get container status \"a41d18d85a4cca4685c5135d5adc65dcebe757c1ac9355d63a50a4660b8f8070\": rpc error: code = NotFound desc = could not find container \"a41d18d85a4cca4685c5135d5adc65dcebe757c1ac9355d63a50a4660b8f8070\": container with ID starting with a41d18d85a4cca4685c5135d5adc65dcebe757c1ac9355d63a50a4660b8f8070 not found: ID does not exist" Jul 07 20:42:10.547587 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:10.547504 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vm4rr\" (UniqueName: \"kubernetes.io/projected/935beb3c-4e53-4a6a-91d3-19acb92f95ff-kube-api-access-vm4rr\") pod \"935beb3c-4e53-4a6a-91d3-19acb92f95ff\" (UID: \"935beb3c-4e53-4a6a-91d3-19acb92f95ff\") " Jul 07 20:42:10.547711 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:10.547692 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/935beb3c-4e53-4a6a-91d3-19acb92f95ff-maas-api-tls\") pod \"935beb3c-4e53-4a6a-91d3-19acb92f95ff\" (UID: \"935beb3c-4e53-4a6a-91d3-19acb92f95ff\") " Jul 07 20:42:10.547767 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:10.547741 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/935beb3c-4e53-4a6a-91d3-19acb92f95ff-openshift-service-ca\") pod \"935beb3c-4e53-4a6a-91d3-19acb92f95ff\" (UID: \"935beb3c-4e53-4a6a-91d3-19acb92f95ff\") " Jul 07 20:42:10.548110 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:10.548084 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/935beb3c-4e53-4a6a-91d3-19acb92f95ff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "935beb3c-4e53-4a6a-91d3-19acb92f95ff" (UID: "935beb3c-4e53-4a6a-91d3-19acb92f95ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 20:42:10.549588 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:10.549565 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/935beb3c-4e53-4a6a-91d3-19acb92f95ff-kube-api-access-vm4rr" (OuterVolumeSpecName: "kube-api-access-vm4rr") pod "935beb3c-4e53-4a6a-91d3-19acb92f95ff" (UID: "935beb3c-4e53-4a6a-91d3-19acb92f95ff"). InnerVolumeSpecName "kube-api-access-vm4rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 20:42:10.549731 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:10.549704 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/935beb3c-4e53-4a6a-91d3-19acb92f95ff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "935beb3c-4e53-4a6a-91d3-19acb92f95ff" (UID: "935beb3c-4e53-4a6a-91d3-19acb92f95ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:42:10.649231 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:10.649200 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/935beb3c-4e53-4a6a-91d3-19acb92f95ff-openshift-service-ca\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:42:10.649231 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:10.649226 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm4rr\" (UniqueName: \"kubernetes.io/projected/935beb3c-4e53-4a6a-91d3-19acb92f95ff-kube-api-access-vm4rr\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:42:10.649231 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:10.649237 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/935beb3c-4e53-4a6a-91d3-19acb92f95ff-maas-api-tls\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:42:10.823361 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:10.823323 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h"] Jul 07 20:42:10.830374 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:10.830347 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-e9f8cf-5779ccb78-qfr2h"] Jul 07 20:42:11.686362 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:11.686327 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="935beb3c-4e53-4a6a-91d3-19acb92f95ff" path="/var/lib/kubelet/pods/935beb3c-4e53-4a6a-91d3-19acb92f95ff/volumes" Jul 07 20:42:23.177528 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:23.177486 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" podUID="c430a278-d7f9-4482-9267-98fab817bb08" containerName="main" containerID="cri-o://aeac4c5f16c357d9f5d5fa116884628128406e8d5be8994aa017f3b3fd98e158" gracePeriod=45 Jul 07 20:42:23.428611 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:23.428544 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" Jul 07 20:42:23.567781 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:23.567745 2565 generic.go:356] "Generic (PLEG): container finished" podID="c430a278-d7f9-4482-9267-98fab817bb08" containerID="aeac4c5f16c357d9f5d5fa116884628128406e8d5be8994aa017f3b3fd98e158" exitCode=0 Jul 07 20:42:23.567950 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:23.567788 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" Jul 07 20:42:23.567950 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:23.567807 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" event={"ID":"c430a278-d7f9-4482-9267-98fab817bb08","Type":"ContainerDied","Data":"aeac4c5f16c357d9f5d5fa116884628128406e8d5be8994aa017f3b3fd98e158"} Jul 07 20:42:23.567950 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:23.567847 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl" event={"ID":"c430a278-d7f9-4482-9267-98fab817bb08","Type":"ContainerDied","Data":"3768d4171777388a17c7bac2616e8e61bdbfba711c72925b6e8f8754cd7228c9"} Jul 07 20:42:23.567950 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:23.567871 2565 scope.go:117] "RemoveContainer" containerID="aeac4c5f16c357d9f5d5fa116884628128406e8d5be8994aa017f3b3fd98e158" Jul 07 20:42:23.575863 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:23.575840 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c430a278-d7f9-4482-9267-98fab817bb08-tls-certs\") pod \"c430a278-d7f9-4482-9267-98fab817bb08\" (UID: \"c430a278-d7f9-4482-9267-98fab817bb08\") " Jul 07 20:42:23.575971 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:23.575912 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqb8m\" (UniqueName: \"kubernetes.io/projected/c430a278-d7f9-4482-9267-98fab817bb08-kube-api-access-pqb8m\") pod \"c430a278-d7f9-4482-9267-98fab817bb08\" (UID: \"c430a278-d7f9-4482-9267-98fab817bb08\") " Jul 07 20:42:23.576035 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:23.576012 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c430a278-d7f9-4482-9267-98fab817bb08-home\") pod \"c430a278-d7f9-4482-9267-98fab817bb08\" (UID: \"c430a278-d7f9-4482-9267-98fab817bb08\") " Jul 07 20:42:23.576117 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:23.576096 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c430a278-d7f9-4482-9267-98fab817bb08-tmp-dir\") pod \"c430a278-d7f9-4482-9267-98fab817bb08\" (UID: \"c430a278-d7f9-4482-9267-98fab817bb08\") " Jul 07 20:42:23.576345 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:23.576147 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c430a278-d7f9-4482-9267-98fab817bb08-dshm\") pod \"c430a278-d7f9-4482-9267-98fab817bb08\" (UID: \"c430a278-d7f9-4482-9267-98fab817bb08\") " Jul 07 20:42:23.576345 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:23.576190 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c430a278-d7f9-4482-9267-98fab817bb08-model-cache\") pod \"c430a278-d7f9-4482-9267-98fab817bb08\" (UID: \"c430a278-d7f9-4482-9267-98fab817bb08\") " Jul 07 20:42:23.576345 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:23.576274 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c430a278-d7f9-4482-9267-98fab817bb08-home" (OuterVolumeSpecName: "home") pod "c430a278-d7f9-4482-9267-98fab817bb08" (UID: "c430a278-d7f9-4482-9267-98fab817bb08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:42:23.576345 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:23.576322 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c430a278-d7f9-4482-9267-98fab817bb08-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c430a278-d7f9-4482-9267-98fab817bb08" (UID: "c430a278-d7f9-4482-9267-98fab817bb08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:42:23.576678 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:23.576432 2565 scope.go:117] "RemoveContainer" containerID="aeac4c5f16c357d9f5d5fa116884628128406e8d5be8994aa017f3b3fd98e158" Jul 07 20:42:23.576678 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:23.576600 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c430a278-d7f9-4482-9267-98fab817bb08-home\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:42:23.576678 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:23.576621 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c430a278-d7f9-4482-9267-98fab817bb08-tmp-dir\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:42:23.576833 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:23.576694 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c430a278-d7f9-4482-9267-98fab817bb08-model-cache" (OuterVolumeSpecName: "model-cache") pod "c430a278-d7f9-4482-9267-98fab817bb08" (UID: "c430a278-d7f9-4482-9267-98fab817bb08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:42:23.576833 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:42:23.576755 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeac4c5f16c357d9f5d5fa116884628128406e8d5be8994aa017f3b3fd98e158\": container with ID starting with aeac4c5f16c357d9f5d5fa116884628128406e8d5be8994aa017f3b3fd98e158 not found: ID does not exist" containerID="aeac4c5f16c357d9f5d5fa116884628128406e8d5be8994aa017f3b3fd98e158" Jul 07 20:42:23.576833 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:23.576780 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeac4c5f16c357d9f5d5fa116884628128406e8d5be8994aa017f3b3fd98e158"} err="failed to get container status \"aeac4c5f16c357d9f5d5fa116884628128406e8d5be8994aa017f3b3fd98e158\": rpc error: code = NotFound desc = could not find container \"aeac4c5f16c357d9f5d5fa116884628128406e8d5be8994aa017f3b3fd98e158\": container with ID starting with aeac4c5f16c357d9f5d5fa116884628128406e8d5be8994aa017f3b3fd98e158 not found: ID does not exist" Jul 07 20:42:23.578289 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:23.578263 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c430a278-d7f9-4482-9267-98fab817bb08-dshm" (OuterVolumeSpecName: "dshm") pod "c430a278-d7f9-4482-9267-98fab817bb08" (UID: "c430a278-d7f9-4482-9267-98fab817bb08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:42:23.578388 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:23.578369 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c430a278-d7f9-4482-9267-98fab817bb08-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c430a278-d7f9-4482-9267-98fab817bb08" (UID: "c430a278-d7f9-4482-9267-98fab817bb08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:42:23.578388 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:23.578381 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c430a278-d7f9-4482-9267-98fab817bb08-kube-api-access-pqb8m" (OuterVolumeSpecName: "kube-api-access-pqb8m") pod "c430a278-d7f9-4482-9267-98fab817bb08" (UID: "c430a278-d7f9-4482-9267-98fab817bb08"). InnerVolumeSpecName "kube-api-access-pqb8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 20:42:23.678049 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:23.678017 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c430a278-d7f9-4482-9267-98fab817bb08-dshm\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:42:23.678207 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:23.678064 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c430a278-d7f9-4482-9267-98fab817bb08-model-cache\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:42:23.678207 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:23.678075 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c430a278-d7f9-4482-9267-98fab817bb08-tls-certs\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:42:23.678207 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:23.678083 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqb8m\" (UniqueName: \"kubernetes.io/projected/c430a278-d7f9-4482-9267-98fab817bb08-kube-api-access-pqb8m\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:42:23.884217 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:23.884184 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl"] Jul 07 20:42:23.888448 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:23.888420 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b1597e/rate-test-model-b1597e-kserve-75bb78ff75-k5qgl"] Jul 07 20:42:25.684900 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:25.684866 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c430a278-d7f9-4482-9267-98fab817bb08" path="/var/lib/kubelet/pods/c430a278-d7f9-4482-9267-98fab817bb08/volumes" Jul 07 20:42:52.254841 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:52.254806 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg"] Jul 07 20:42:52.255470 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:52.255063 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" podUID="09df4297-042d-4762-b66a-3dab2e909d45" containerName="istio-proxy" containerID="cri-o://a66f5f0fc9b36b2b565a13e5750bcbfbf554d5ade91b7133986bc5676ff31911" gracePeriod=30 Jul 07 20:42:53.492393 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:53.492359 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb"] Jul 07 20:42:53.492853 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:53.492599 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb" podUID="9983b52f-3826-4ed1-906f-1140af31d129" containerName="maas-api" containerID="cri-o://4219c7ad1cdde3d7937978da3dd6dbee0b8d4c86f600fe2caa2f913f6b639fdd" gracePeriod=30 Jul 07 20:42:53.715153 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:53.715116 2565 generic.go:356] "Generic (PLEG): container finished" podID="9983b52f-3826-4ed1-906f-1140af31d129" containerID="4219c7ad1cdde3d7937978da3dd6dbee0b8d4c86f600fe2caa2f913f6b639fdd" exitCode=0 Jul 07 20:42:53.715306 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:53.715198 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb" event={"ID":"9983b52f-3826-4ed1-906f-1140af31d129","Type":"ContainerDied","Data":"4219c7ad1cdde3d7937978da3dd6dbee0b8d4c86f600fe2caa2f913f6b639fdd"} Jul 07 20:42:53.740849 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:53.740817 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb" Jul 07 20:42:53.786596 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:53.786517 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9983b52f-3826-4ed1-906f-1140af31d129-maas-api-tls\") pod \"9983b52f-3826-4ed1-906f-1140af31d129\" (UID: \"9983b52f-3826-4ed1-906f-1140af31d129\") " Jul 07 20:42:53.786596 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:53.786580 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvdmh\" (UniqueName: \"kubernetes.io/projected/9983b52f-3826-4ed1-906f-1140af31d129-kube-api-access-dvdmh\") pod \"9983b52f-3826-4ed1-906f-1140af31d129\" (UID: \"9983b52f-3826-4ed1-906f-1140af31d129\") " Jul 07 20:42:53.786833 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:53.786622 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9983b52f-3826-4ed1-906f-1140af31d129-openshift-service-ca\") pod \"9983b52f-3826-4ed1-906f-1140af31d129\" (UID: \"9983b52f-3826-4ed1-906f-1140af31d129\") " Jul 07 20:42:53.787041 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:53.787017 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9983b52f-3826-4ed1-906f-1140af31d129-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9983b52f-3826-4ed1-906f-1140af31d129" (UID: "9983b52f-3826-4ed1-906f-1140af31d129"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 20:42:53.788581 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:53.788558 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9983b52f-3826-4ed1-906f-1140af31d129-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9983b52f-3826-4ed1-906f-1140af31d129" (UID: "9983b52f-3826-4ed1-906f-1140af31d129"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 20:42:53.788757 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:53.788740 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9983b52f-3826-4ed1-906f-1140af31d129-kube-api-access-dvdmh" (OuterVolumeSpecName: "kube-api-access-dvdmh") pod "9983b52f-3826-4ed1-906f-1140af31d129" (UID: "9983b52f-3826-4ed1-906f-1140af31d129"). InnerVolumeSpecName "kube-api-access-dvdmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 20:42:53.887996 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:53.887952 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9983b52f-3826-4ed1-906f-1140af31d129-maas-api-tls\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:42:53.887996 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:53.887984 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvdmh\" (UniqueName: \"kubernetes.io/projected/9983b52f-3826-4ed1-906f-1140af31d129-kube-api-access-dvdmh\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:42:53.887996 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:53.887995 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9983b52f-3826-4ed1-906f-1140af31d129-openshift-service-ca\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:42:54.722429 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:54.722367 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb" Jul 07 20:42:54.722904 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:54.722427 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb" event={"ID":"9983b52f-3826-4ed1-906f-1140af31d129","Type":"ContainerDied","Data":"73547ae9d41a6bfe959bfc350db927bfbcd54e23e135d817d706260f5c4a4abd"} Jul 07 20:42:54.722904 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:54.722495 2565 scope.go:117] "RemoveContainer" containerID="4219c7ad1cdde3d7937978da3dd6dbee0b8d4c86f600fe2caa2f913f6b639fdd" Jul 07 20:42:54.743821 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:54.743792 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb"] Jul 07 20:42:54.749599 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:54.749574 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-b1597e-746bbc7cb9-n5nsb"] Jul 07 20:42:54.975102 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:54.975018 2565 patch_prober.go:28] interesting pod/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 20:42:54.975102 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:54.975077 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" podUID="09df4297-042d-4762-b66a-3dab2e909d45" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 20:42:55.684614 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:55.684579 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9983b52f-3826-4ed1-906f-1140af31d129" path="/var/lib/kubelet/pods/9983b52f-3826-4ed1-906f-1140af31d129/volumes" Jul 07 20:42:57.518599 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.518527 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:42:57.624413 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.624368 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09df4297-042d-4762-b66a-3dab2e909d45-istio-data\") pod \"09df4297-042d-4762-b66a-3dab2e909d45\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " Jul 07 20:42:57.624594 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.624472 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09df4297-042d-4762-b66a-3dab2e909d45-workload-socket\") pod \"09df4297-042d-4762-b66a-3dab2e909d45\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " Jul 07 20:42:57.624594 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.624499 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09df4297-042d-4762-b66a-3dab2e909d45-istiod-ca-cert\") pod \"09df4297-042d-4762-b66a-3dab2e909d45\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " Jul 07 20:42:57.624594 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.624566 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09df4297-042d-4762-b66a-3dab2e909d45-istio-envoy\") pod \"09df4297-042d-4762-b66a-3dab2e909d45\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " Jul 07 20:42:57.624731 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.624622 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09df4297-042d-4762-b66a-3dab2e909d45-workload-certs\") pod \"09df4297-042d-4762-b66a-3dab2e909d45\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " Jul 07 20:42:57.624731 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.624686 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09df4297-042d-4762-b66a-3dab2e909d45-istio-token\") pod \"09df4297-042d-4762-b66a-3dab2e909d45\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " Jul 07 20:42:57.624830 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.624732 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gc69\" (UniqueName: \"kubernetes.io/projected/09df4297-042d-4762-b66a-3dab2e909d45-kube-api-access-6gc69\") pod \"09df4297-042d-4762-b66a-3dab2e909d45\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " Jul 07 20:42:57.624830 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.624788 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09df4297-042d-4762-b66a-3dab2e909d45-credential-socket\") pod \"09df4297-042d-4762-b66a-3dab2e909d45\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " Jul 07 20:42:57.624830 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.624783 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09df4297-042d-4762-b66a-3dab2e909d45-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "09df4297-042d-4762-b66a-3dab2e909d45" (UID: "09df4297-042d-4762-b66a-3dab2e909d45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:42:57.624830 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.624818 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09df4297-042d-4762-b66a-3dab2e909d45-istio-podinfo\") pod \"09df4297-042d-4762-b66a-3dab2e909d45\" (UID: \"09df4297-042d-4762-b66a-3dab2e909d45\") " Jul 07 20:42:57.625040 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.624959 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09df4297-042d-4762-b66a-3dab2e909d45-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "09df4297-042d-4762-b66a-3dab2e909d45" (UID: "09df4297-042d-4762-b66a-3dab2e909d45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:42:57.625040 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.624995 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09df4297-042d-4762-b66a-3dab2e909d45-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "09df4297-042d-4762-b66a-3dab2e909d45" (UID: "09df4297-042d-4762-b66a-3dab2e909d45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 20:42:57.625166 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.625129 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09df4297-042d-4762-b66a-3dab2e909d45-workload-socket\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:42:57.625166 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.625144 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09df4297-042d-4762-b66a-3dab2e909d45-istiod-ca-cert\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:42:57.625166 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.625153 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09df4297-042d-4762-b66a-3dab2e909d45-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "09df4297-042d-4762-b66a-3dab2e909d45" (UID: "09df4297-042d-4762-b66a-3dab2e909d45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:42:57.625312 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.625159 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09df4297-042d-4762-b66a-3dab2e909d45-workload-certs\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:42:57.625312 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.625260 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09df4297-042d-4762-b66a-3dab2e909d45-istio-data" (OuterVolumeSpecName: "istio-data") pod "09df4297-042d-4762-b66a-3dab2e909d45" (UID: "09df4297-042d-4762-b66a-3dab2e909d45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:42:57.626910 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.626879 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09df4297-042d-4762-b66a-3dab2e909d45-istio-token" (OuterVolumeSpecName: "istio-token") pod "09df4297-042d-4762-b66a-3dab2e909d45" (UID: "09df4297-042d-4762-b66a-3dab2e909d45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 20:42:57.627025 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.627003 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09df4297-042d-4762-b66a-3dab2e909d45-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "09df4297-042d-4762-b66a-3dab2e909d45" (UID: "09df4297-042d-4762-b66a-3dab2e909d45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 20:42:57.627308 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.627273 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09df4297-042d-4762-b66a-3dab2e909d45-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "09df4297-042d-4762-b66a-3dab2e909d45" (UID: "09df4297-042d-4762-b66a-3dab2e909d45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 20:42:57.627308 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.627289 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09df4297-042d-4762-b66a-3dab2e909d45-kube-api-access-6gc69" (OuterVolumeSpecName: "kube-api-access-6gc69") pod "09df4297-042d-4762-b66a-3dab2e909d45" (UID: "09df4297-042d-4762-b66a-3dab2e909d45"). InnerVolumeSpecName "kube-api-access-6gc69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 20:42:57.725835 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.725801 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09df4297-042d-4762-b66a-3dab2e909d45-istio-podinfo\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:42:57.725835 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.725830 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09df4297-042d-4762-b66a-3dab2e909d45-istio-data\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:42:57.725835 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.725839 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09df4297-042d-4762-b66a-3dab2e909d45-istio-envoy\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:42:57.725835 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.725846 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09df4297-042d-4762-b66a-3dab2e909d45-istio-token\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:42:57.726095 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.725854 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gc69\" (UniqueName: \"kubernetes.io/projected/09df4297-042d-4762-b66a-3dab2e909d45-kube-api-access-6gc69\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:42:57.726095 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.725864 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09df4297-042d-4762-b66a-3dab2e909d45-credential-socket\") on node \"ip-10-0-133-229.ec2.internal\" DevicePath \"\"" Jul 07 20:42:57.739223 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.739191 2565 generic.go:356] "Generic (PLEG): container finished" podID="09df4297-042d-4762-b66a-3dab2e909d45" containerID="a66f5f0fc9b36b2b565a13e5750bcbfbf554d5ade91b7133986bc5676ff31911" exitCode=0 Jul 07 20:42:57.739367 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.739244 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" Jul 07 20:42:57.739367 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.739237 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" event={"ID":"09df4297-042d-4762-b66a-3dab2e909d45","Type":"ContainerDied","Data":"a66f5f0fc9b36b2b565a13e5750bcbfbf554d5ade91b7133986bc5676ff31911"} Jul 07 20:42:57.739533 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.739376 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg" event={"ID":"09df4297-042d-4762-b66a-3dab2e909d45","Type":"ContainerDied","Data":"c45d61a0af88787d0e67992bd8dbc01e3911d68369520ddeff73ea1fc918cddd"} Jul 07 20:42:57.739533 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.739435 2565 scope.go:117] "RemoveContainer" containerID="a66f5f0fc9b36b2b565a13e5750bcbfbf554d5ade91b7133986bc5676ff31911" Jul 07 20:42:57.748437 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.748414 2565 scope.go:117] "RemoveContainer" containerID="a86ce99583e7a6042ecc2d463604c7d65dbeb16ecf5d4d45cdbd9a5325909d75" Jul 07 20:42:57.756712 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.756692 2565 scope.go:117] "RemoveContainer" containerID="a66f5f0fc9b36b2b565a13e5750bcbfbf554d5ade91b7133986bc5676ff31911" Jul 07 20:42:57.756993 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:42:57.756972 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a66f5f0fc9b36b2b565a13e5750bcbfbf554d5ade91b7133986bc5676ff31911\": container with ID starting with a66f5f0fc9b36b2b565a13e5750bcbfbf554d5ade91b7133986bc5676ff31911 not found: ID does not exist" containerID="a66f5f0fc9b36b2b565a13e5750bcbfbf554d5ade91b7133986bc5676ff31911" Jul 07 20:42:57.757064 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.757005 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a66f5f0fc9b36b2b565a13e5750bcbfbf554d5ade91b7133986bc5676ff31911"} err="failed to get container status \"a66f5f0fc9b36b2b565a13e5750bcbfbf554d5ade91b7133986bc5676ff31911\": rpc error: code = NotFound desc = could not find container \"a66f5f0fc9b36b2b565a13e5750bcbfbf554d5ade91b7133986bc5676ff31911\": container with ID starting with a66f5f0fc9b36b2b565a13e5750bcbfbf554d5ade91b7133986bc5676ff31911 not found: ID does not exist" Jul 07 20:42:57.757064 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.757030 2565 scope.go:117] "RemoveContainer" containerID="a86ce99583e7a6042ecc2d463604c7d65dbeb16ecf5d4d45cdbd9a5325909d75" Jul 07 20:42:57.757257 ip-10-0-133-229 kubenswrapper[2565]: E0707 20:42:57.757239 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a86ce99583e7a6042ecc2d463604c7d65dbeb16ecf5d4d45cdbd9a5325909d75\": container with ID starting with a86ce99583e7a6042ecc2d463604c7d65dbeb16ecf5d4d45cdbd9a5325909d75 not found: ID does not exist" containerID="a86ce99583e7a6042ecc2d463604c7d65dbeb16ecf5d4d45cdbd9a5325909d75" Jul 07 20:42:57.757303 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.757264 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a86ce99583e7a6042ecc2d463604c7d65dbeb16ecf5d4d45cdbd9a5325909d75"} err="failed to get container status \"a86ce99583e7a6042ecc2d463604c7d65dbeb16ecf5d4d45cdbd9a5325909d75\": rpc error: code = NotFound desc = could not find container \"a86ce99583e7a6042ecc2d463604c7d65dbeb16ecf5d4d45cdbd9a5325909d75\": container with ID starting with a86ce99583e7a6042ecc2d463604c7d65dbeb16ecf5d4d45cdbd9a5325909d75 not found: ID does not exist" Jul 07 20:42:57.758253 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.758236 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg"] Jul 07 20:42:57.764472 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:57.764450 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-e9f8cf-openshift-default-66cd9b86b7-gsvpg"] Jul 07 20:42:59.685301 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:42:59.685266 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09df4297-042d-4762-b66a-3dab2e909d45" path="/var/lib/kubelet/pods/09df4297-042d-4762-b66a-3dab2e909d45/volumes" Jul 07 20:43:05.965388 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:05.965357 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-77476c4d87-twlcf_ce7010ee-0687-4868-9077-7dfdd7a3545e/maas-api/0.log" Jul 07 20:43:06.088167 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:06.088131 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724255-kzznf_8ee5a14b-25d4-40c5-b111-7d5f89653d34/cleanup/0.log" Jul 07 20:43:06.207552 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:06.207522 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724270-qh4mr_0e431ffc-713b-41c8-bd1a-377110182a1c/cleanup/0.log" Jul 07 20:43:06.835483 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:06.835454 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-659495cbcb-pw9cc_510a6e1c-e835-4397-9f70-e1fe37af3f2b/manager/0.log" Jul 07 20:43:06.842524 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:06.842476 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-659495cbcb-pw9cc_510a6e1c-e835-4397-9f70-e1fe37af3f2b/copy-manifests/0.log" Jul 07 20:43:08.845028 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:08.845000 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-h7wf9_83bb2bf4-0574-43a0-80a2-43560f286fc7/kuadrant-console-plugin/0.log" Jul 07 20:43:09.350311 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:09.350278 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-j69cz_e1085214-ede2-463c-abd1-74a74cd10cb0/manager/0.log" Jul 07 20:43:09.980691 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:09.980659 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68f4c445fc-48crl_18992f4a-342c-4bd6-81fb-5e5fc9d951d1/kube-auth-proxy/0.log" Jul 07 20:43:11.006122 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:11.006089 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-2fpj2_14085b5e-a02b-43d3-b966-4f777cf80c2c/main/0.log" Jul 07 20:43:11.383223 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:11.383194 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgng8w_c27ba098-46ba-4167-bedd-da6f0a16085e/main/0.log" Jul 07 20:43:16.622024 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:16.621991 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ww672/must-gather-drfdb"] Jul 07 20:43:16.622550 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:16.622530 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="935beb3c-4e53-4a6a-91d3-19acb92f95ff" containerName="maas-api" Jul 07 20:43:16.622657 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:16.622554 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="935beb3c-4e53-4a6a-91d3-19acb92f95ff" containerName="maas-api" Jul 07 20:43:16.622657 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:16.622592 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09df4297-042d-4762-b66a-3dab2e909d45" containerName="istio-proxy" Jul 07 20:43:16.622657 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:16.622600 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="09df4297-042d-4762-b66a-3dab2e909d45" containerName="istio-proxy" Jul 07 20:43:16.622657 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:16.622630 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09df4297-042d-4762-b66a-3dab2e909d45" containerName="istio-proxy" Jul 07 20:43:16.622657 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:16.622636 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="09df4297-042d-4762-b66a-3dab2e909d45" containerName="istio-proxy" Jul 07 20:43:16.622657 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:16.622642 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c430a278-d7f9-4482-9267-98fab817bb08" containerName="main" Jul 07 20:43:16.622657 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:16.622648 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c430a278-d7f9-4482-9267-98fab817bb08" containerName="main" Jul 07 20:43:16.622657 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:16.622657 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9983b52f-3826-4ed1-906f-1140af31d129" containerName="maas-api" Jul 07 20:43:16.622657 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:16.622662 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9983b52f-3826-4ed1-906f-1140af31d129" containerName="maas-api" Jul 07 20:43:16.622973 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:16.622734 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="09df4297-042d-4762-b66a-3dab2e909d45" containerName="istio-proxy" Jul 07 20:43:16.622973 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:16.622747 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="935beb3c-4e53-4a6a-91d3-19acb92f95ff" containerName="maas-api" Jul 07 20:43:16.622973 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:16.622753 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9983b52f-3826-4ed1-906f-1140af31d129" containerName="maas-api" Jul 07 20:43:16.622973 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:16.622759 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="09df4297-042d-4762-b66a-3dab2e909d45" containerName="istio-proxy" Jul 07 20:43:16.622973 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:16.622766 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c430a278-d7f9-4482-9267-98fab817bb08" containerName="main" Jul 07 20:43:16.627735 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:16.627719 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ww672/must-gather-drfdb" Jul 07 20:43:16.630180 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:16.630159 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ww672\"/\"openshift-service-ca.crt\"" Jul 07 20:43:16.631264 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:16.631236 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ww672\"/\"kube-root-ca.crt\"" Jul 07 20:43:16.631264 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:16.631246 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ww672\"/\"default-dockercfg-k44j8\"" Jul 07 20:43:16.634135 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:16.634115 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ww672/must-gather-drfdb"] Jul 07 20:43:16.703626 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:16.703595 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2whb7\" (UniqueName: \"kubernetes.io/projected/bd4ee6e5-e7d7-4714-bf6b-e555e694219a-kube-api-access-2whb7\") pod \"must-gather-drfdb\" (UID: \"bd4ee6e5-e7d7-4714-bf6b-e555e694219a\") " pod="openshift-must-gather-ww672/must-gather-drfdb" Jul 07 20:43:16.703795 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:16.703653 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bd4ee6e5-e7d7-4714-bf6b-e555e694219a-must-gather-output\") pod \"must-gather-drfdb\" (UID: \"bd4ee6e5-e7d7-4714-bf6b-e555e694219a\") " pod="openshift-must-gather-ww672/must-gather-drfdb" Jul 07 20:43:16.804194 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:16.804157 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2whb7\" (UniqueName: \"kubernetes.io/projected/bd4ee6e5-e7d7-4714-bf6b-e555e694219a-kube-api-access-2whb7\") pod \"must-gather-drfdb\" (UID: \"bd4ee6e5-e7d7-4714-bf6b-e555e694219a\") " pod="openshift-must-gather-ww672/must-gather-drfdb" Jul 07 20:43:16.804356 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:16.804317 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bd4ee6e5-e7d7-4714-bf6b-e555e694219a-must-gather-output\") pod \"must-gather-drfdb\" (UID: \"bd4ee6e5-e7d7-4714-bf6b-e555e694219a\") " pod="openshift-must-gather-ww672/must-gather-drfdb" Jul 07 20:43:16.804700 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:16.804677 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bd4ee6e5-e7d7-4714-bf6b-e555e694219a-must-gather-output\") pod \"must-gather-drfdb\" (UID: \"bd4ee6e5-e7d7-4714-bf6b-e555e694219a\") " pod="openshift-must-gather-ww672/must-gather-drfdb" Jul 07 20:43:16.813004 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:16.812983 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2whb7\" (UniqueName: \"kubernetes.io/projected/bd4ee6e5-e7d7-4714-bf6b-e555e694219a-kube-api-access-2whb7\") pod \"must-gather-drfdb\" (UID: \"bd4ee6e5-e7d7-4714-bf6b-e555e694219a\") " pod="openshift-must-gather-ww672/must-gather-drfdb" Jul 07 20:43:16.937767 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:16.937740 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ww672/must-gather-drfdb" Jul 07 20:43:17.090898 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:17.090869 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ww672/must-gather-drfdb"] Jul 07 20:43:17.094447 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:43:17.094390 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd4ee6e5_e7d7_4714_bf6b_e555e694219a.slice/crio-70fb1a86a1e4da061a78e22c31e9246ef32905278410e7bed715f949caa8a117 WatchSource:0}: Error finding container 70fb1a86a1e4da061a78e22c31e9246ef32905278410e7bed715f949caa8a117: Status 404 returned error can't find the container with id 70fb1a86a1e4da061a78e22c31e9246ef32905278410e7bed715f949caa8a117 Jul 07 20:43:17.096472 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:17.096454 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 20:43:17.839895 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:17.839865 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ww672/must-gather-drfdb" event={"ID":"bd4ee6e5-e7d7-4714-bf6b-e555e694219a","Type":"ContainerStarted","Data":"70fb1a86a1e4da061a78e22c31e9246ef32905278410e7bed715f949caa8a117"} Jul 07 20:43:18.847837 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:18.847797 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ww672/must-gather-drfdb" event={"ID":"bd4ee6e5-e7d7-4714-bf6b-e555e694219a","Type":"ContainerStarted","Data":"e405826ee5358695378410aa81e1adf5bfd45fc6a7aafd7a36928c2e8d94b325"} Jul 07 20:43:18.848241 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:18.847848 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ww672/must-gather-drfdb" event={"ID":"bd4ee6e5-e7d7-4714-bf6b-e555e694219a","Type":"ContainerStarted","Data":"38c07aa026d85af2ed128082699d4a1872491f823ad0a00d0654818bd3a9e696"} Jul 07 20:43:18.867163 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:18.867097 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ww672/must-gather-drfdb" podStartSLOduration=1.76712523 podStartE2EDuration="2.867075235s" totalImagesPullingTime="1.099950005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 20:43:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:43:18.86560415 +0000 UTC m=+2519.020066100" watchObservedRunningTime="2026-07-07 20:43:18.867075235 +0000 UTC m=+2519.021537185" Jul 07 20:43:19.916651 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:19.916614 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zr7c2_17142129-e574-4092-bd8d-0ffa3de6507a/global-pull-secret-syncer/0.log" Jul 07 20:43:20.008888 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:20.008853 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jlj7d_2c31e770-b420-42cf-8bf6-a7f8540ed592/konnectivity-agent/0.log" Jul 07 20:43:20.065756 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:20.065724 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-229.ec2.internal_59bf4ed301a47b79107844633c8ed6ad/haproxy/0.log" Jul 07 20:43:25.768120 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:25.767721 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-h7wf9_83bb2bf4-0574-43a0-80a2-43560f286fc7/kuadrant-console-plugin/0.log" Jul 07 20:43:26.100450 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:26.100351 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-j69cz_e1085214-ede2-463c-abd1-74a74cd10cb0/manager/0.log" Jul 07 20:43:27.720953 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:27.720866 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3f440cf2-76c9-4344-bffc-365865c94c5f/alertmanager/0.log" Jul 07 20:43:27.744875 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:27.744822 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3f440cf2-76c9-4344-bffc-365865c94c5f/config-reloader/0.log" Jul 07 20:43:27.769983 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:27.769929 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3f440cf2-76c9-4344-bffc-365865c94c5f/kube-rbac-proxy-web/0.log" Jul 07 20:43:27.792982 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:27.792956 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3f440cf2-76c9-4344-bffc-365865c94c5f/kube-rbac-proxy/0.log" Jul 07 20:43:27.818612 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:27.818582 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3f440cf2-76c9-4344-bffc-365865c94c5f/kube-rbac-proxy-metric/0.log" Jul 07 20:43:27.850465 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:27.850353 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3f440cf2-76c9-4344-bffc-365865c94c5f/prom-label-proxy/0.log" Jul 07 20:43:27.873809 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:27.873776 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3f440cf2-76c9-4344-bffc-365865c94c5f/init-config-reloader/0.log" Jul 07 20:43:27.951360 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:27.951328 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-bsdc6_1216713e-ac05-4a9b-94d0-5ec9e0caad6a/kube-state-metrics/0.log" Jul 07 20:43:27.972865 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:27.972788 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-bsdc6_1216713e-ac05-4a9b-94d0-5ec9e0caad6a/kube-rbac-proxy-main/0.log" Jul 07 20:43:27.996954 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:27.996925 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-bsdc6_1216713e-ac05-4a9b-94d0-5ec9e0caad6a/kube-rbac-proxy-self/0.log" Jul 07 20:43:28.032579 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:28.032538 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-76d99668b4-2x4pv_6eede5f7-90c3-4d64-a6bc-ff8580fb3a33/metrics-server/0.log" Jul 07 20:43:28.057911 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:28.057874 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-nd79v_7301ae59-b24d-4bda-a5c0-7b8036966f8e/monitoring-plugin/0.log" Jul 07 20:43:28.087923 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:28.087901 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-89snw_d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18/node-exporter/0.log" Jul 07 20:43:28.112054 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:28.112027 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-89snw_d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18/kube-rbac-proxy/0.log" Jul 07 20:43:28.136712 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:28.136681 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-89snw_d57ed5e1-7eb2-41c1-bcf7-edcd986e2a18/init-textfile/0.log" Jul 07 20:43:28.332342 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:28.332308 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-bsssh_cab569a8-8055-4e0f-bba9-c698e829a799/kube-rbac-proxy-main/0.log" Jul 07 20:43:28.360456 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:28.360424 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-bsssh_cab569a8-8055-4e0f-bba9-c698e829a799/kube-rbac-proxy-self/0.log" Jul 07 20:43:28.385640 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:28.385610 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-bsssh_cab569a8-8055-4e0f-bba9-c698e829a799/openshift-state-metrics/0.log" Jul 07 20:43:28.419214 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:28.419183 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_535f2334-abec-4253-a96b-ea6f4d63a585/prometheus/0.log" Jul 07 20:43:28.440908 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:28.440884 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_535f2334-abec-4253-a96b-ea6f4d63a585/config-reloader/0.log" Jul 07 20:43:28.469477 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:28.469449 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_535f2334-abec-4253-a96b-ea6f4d63a585/thanos-sidecar/0.log" Jul 07 20:43:28.494871 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:28.494787 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_535f2334-abec-4253-a96b-ea6f4d63a585/kube-rbac-proxy-web/0.log" Jul 07 20:43:28.516726 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:28.516696 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_535f2334-abec-4253-a96b-ea6f4d63a585/kube-rbac-proxy/0.log" Jul 07 20:43:28.540556 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:28.540527 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_535f2334-abec-4253-a96b-ea6f4d63a585/kube-rbac-proxy-thanos/0.log" Jul 07 20:43:28.564885 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:28.564846 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_535f2334-abec-4253-a96b-ea6f4d63a585/init-config-reloader/0.log" Jul 07 20:43:28.598976 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:28.598940 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-2vmh4_eb073572-f7a1-40e6-9485-a6206bffda9b/prometheus-operator/0.log" Jul 07 20:43:28.620938 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:28.620900 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-2vmh4_eb073572-f7a1-40e6-9485-a6206bffda9b/kube-rbac-proxy/0.log" Jul 07 20:43:28.681908 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:28.681839 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-58fdf88bf5-9kzj9_333927dd-8b21-49c8-b36a-8050642aad71/telemeter-client/0.log" Jul 07 20:43:28.706160 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:28.706110 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-58fdf88bf5-9kzj9_333927dd-8b21-49c8-b36a-8050642aad71/reload/0.log" Jul 07 20:43:28.729959 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:28.729930 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-58fdf88bf5-9kzj9_333927dd-8b21-49c8-b36a-8050642aad71/kube-rbac-proxy/0.log" Jul 07 20:43:28.765046 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:28.764968 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-677c97cfd9-8jl49_1a5cb3fd-134b-4b74-b687-8b05bdeb31b6/thanos-query/0.log" Jul 07 20:43:28.788098 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:28.788067 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-677c97cfd9-8jl49_1a5cb3fd-134b-4b74-b687-8b05bdeb31b6/kube-rbac-proxy-web/0.log" Jul 07 20:43:28.812378 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:28.812346 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-677c97cfd9-8jl49_1a5cb3fd-134b-4b74-b687-8b05bdeb31b6/kube-rbac-proxy/0.log" Jul 07 20:43:28.834468 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:28.834441 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-677c97cfd9-8jl49_1a5cb3fd-134b-4b74-b687-8b05bdeb31b6/prom-label-proxy/0.log" Jul 07 20:43:28.859634 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:28.859593 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-677c97cfd9-8jl49_1a5cb3fd-134b-4b74-b687-8b05bdeb31b6/kube-rbac-proxy-rules/0.log" Jul 07 20:43:28.884120 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:28.884077 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-677c97cfd9-8jl49_1a5cb3fd-134b-4b74-b687-8b05bdeb31b6/kube-rbac-proxy-metrics/0.log" Jul 07 20:43:29.977208 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:29.977169 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ww672/perf-node-gather-daemonset-kgk2t"] Jul 07 20:43:29.979806 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:29.979783 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ww672/perf-node-gather-daemonset-kgk2t" Jul 07 20:43:29.994197 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:29.994172 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ww672/perf-node-gather-daemonset-kgk2t"] Jul 07 20:43:30.039241 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:30.039202 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/792a5e01-37ad-4c7d-9a53-9c40f82d793f-podres\") pod \"perf-node-gather-daemonset-kgk2t\" (UID: \"792a5e01-37ad-4c7d-9a53-9c40f82d793f\") " pod="openshift-must-gather-ww672/perf-node-gather-daemonset-kgk2t" Jul 07 20:43:30.039562 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:30.039538 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/792a5e01-37ad-4c7d-9a53-9c40f82d793f-proc\") pod \"perf-node-gather-daemonset-kgk2t\" (UID: \"792a5e01-37ad-4c7d-9a53-9c40f82d793f\") " pod="openshift-must-gather-ww672/perf-node-gather-daemonset-kgk2t" Jul 07 20:43:30.039698 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:30.039684 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/792a5e01-37ad-4c7d-9a53-9c40f82d793f-sys\") pod \"perf-node-gather-daemonset-kgk2t\" (UID: \"792a5e01-37ad-4c7d-9a53-9c40f82d793f\") " pod="openshift-must-gather-ww672/perf-node-gather-daemonset-kgk2t" Jul 07 20:43:30.039814 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:30.039799 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/792a5e01-37ad-4c7d-9a53-9c40f82d793f-lib-modules\") pod \"perf-node-gather-daemonset-kgk2t\" (UID: \"792a5e01-37ad-4c7d-9a53-9c40f82d793f\") " pod="openshift-must-gather-ww672/perf-node-gather-daemonset-kgk2t" Jul 07 20:43:30.039976 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:30.039960 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xjf9\" (UniqueName: \"kubernetes.io/projected/792a5e01-37ad-4c7d-9a53-9c40f82d793f-kube-api-access-8xjf9\") pod \"perf-node-gather-daemonset-kgk2t\" (UID: \"792a5e01-37ad-4c7d-9a53-9c40f82d793f\") " pod="openshift-must-gather-ww672/perf-node-gather-daemonset-kgk2t" Jul 07 20:43:30.140752 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:30.140700 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/792a5e01-37ad-4c7d-9a53-9c40f82d793f-sys\") pod \"perf-node-gather-daemonset-kgk2t\" (UID: \"792a5e01-37ad-4c7d-9a53-9c40f82d793f\") " pod="openshift-must-gather-ww672/perf-node-gather-daemonset-kgk2t" Jul 07 20:43:30.140979 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:30.140762 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/792a5e01-37ad-4c7d-9a53-9c40f82d793f-lib-modules\") pod \"perf-node-gather-daemonset-kgk2t\" (UID: \"792a5e01-37ad-4c7d-9a53-9c40f82d793f\") " pod="openshift-must-gather-ww672/perf-node-gather-daemonset-kgk2t" Jul 07 20:43:30.140979 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:30.140839 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/792a5e01-37ad-4c7d-9a53-9c40f82d793f-sys\") pod \"perf-node-gather-daemonset-kgk2t\" (UID: \"792a5e01-37ad-4c7d-9a53-9c40f82d793f\") " pod="openshift-must-gather-ww672/perf-node-gather-daemonset-kgk2t" Jul 07 20:43:30.140979 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:30.140843 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xjf9\" (UniqueName: \"kubernetes.io/projected/792a5e01-37ad-4c7d-9a53-9c40f82d793f-kube-api-access-8xjf9\") pod \"perf-node-gather-daemonset-kgk2t\" (UID: \"792a5e01-37ad-4c7d-9a53-9c40f82d793f\") " pod="openshift-must-gather-ww672/perf-node-gather-daemonset-kgk2t" Jul 07 20:43:30.140979 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:30.140935 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/792a5e01-37ad-4c7d-9a53-9c40f82d793f-lib-modules\") pod \"perf-node-gather-daemonset-kgk2t\" (UID: \"792a5e01-37ad-4c7d-9a53-9c40f82d793f\") " pod="openshift-must-gather-ww672/perf-node-gather-daemonset-kgk2t" Jul 07 20:43:30.140979 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:30.140961 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/792a5e01-37ad-4c7d-9a53-9c40f82d793f-podres\") pod \"perf-node-gather-daemonset-kgk2t\" (UID: \"792a5e01-37ad-4c7d-9a53-9c40f82d793f\") " pod="openshift-must-gather-ww672/perf-node-gather-daemonset-kgk2t" Jul 07 20:43:30.141373 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:30.141030 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/792a5e01-37ad-4c7d-9a53-9c40f82d793f-proc\") pod \"perf-node-gather-daemonset-kgk2t\" (UID: \"792a5e01-37ad-4c7d-9a53-9c40f82d793f\") " pod="openshift-must-gather-ww672/perf-node-gather-daemonset-kgk2t" Jul 07 20:43:30.141373 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:30.141085 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/792a5e01-37ad-4c7d-9a53-9c40f82d793f-podres\") pod \"perf-node-gather-daemonset-kgk2t\" (UID: \"792a5e01-37ad-4c7d-9a53-9c40f82d793f\") " pod="openshift-must-gather-ww672/perf-node-gather-daemonset-kgk2t" Jul 07 20:43:30.141373 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:30.141120 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/792a5e01-37ad-4c7d-9a53-9c40f82d793f-proc\") pod \"perf-node-gather-daemonset-kgk2t\" (UID: \"792a5e01-37ad-4c7d-9a53-9c40f82d793f\") " pod="openshift-must-gather-ww672/perf-node-gather-daemonset-kgk2t" Jul 07 20:43:30.149842 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:30.149804 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xjf9\" (UniqueName: \"kubernetes.io/projected/792a5e01-37ad-4c7d-9a53-9c40f82d793f-kube-api-access-8xjf9\") pod \"perf-node-gather-daemonset-kgk2t\" (UID: \"792a5e01-37ad-4c7d-9a53-9c40f82d793f\") " pod="openshift-must-gather-ww672/perf-node-gather-daemonset-kgk2t" Jul 07 20:43:30.294061 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:30.293981 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ww672/perf-node-gather-daemonset-kgk2t" Jul 07 20:43:30.475210 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:30.475175 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ww672/perf-node-gather-daemonset-kgk2t"] Jul 07 20:43:30.477951 ip-10-0-133-229 kubenswrapper[2565]: W0707 20:43:30.477916 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod792a5e01_37ad_4c7d_9a53_9c40f82d793f.slice/crio-0f4932d6f42c7583d92db672d4ad4a15a9aeae7a8391d9bed8c36cb5b657a923 WatchSource:0}: Error finding container 0f4932d6f42c7583d92db672d4ad4a15a9aeae7a8391d9bed8c36cb5b657a923: Status 404 returned error can't find the container with id 0f4932d6f42c7583d92db672d4ad4a15a9aeae7a8391d9bed8c36cb5b657a923 Jul 07 20:43:30.941342 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:30.941302 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ww672/perf-node-gather-daemonset-kgk2t" event={"ID":"792a5e01-37ad-4c7d-9a53-9c40f82d793f","Type":"ContainerStarted","Data":"3f6a8ce91452e5761fa30e46c66ad903639b68f11d5f6ae70dab9b0cc0c64362"} Jul 07 20:43:30.941551 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:30.941441 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ww672/perf-node-gather-daemonset-kgk2t" event={"ID":"792a5e01-37ad-4c7d-9a53-9c40f82d793f","Type":"ContainerStarted","Data":"0f4932d6f42c7583d92db672d4ad4a15a9aeae7a8391d9bed8c36cb5b657a923"} Jul 07 20:43:30.941551 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:30.941478 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ww672/perf-node-gather-daemonset-kgk2t" Jul 07 20:43:30.959431 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:30.959132 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ww672/perf-node-gather-daemonset-kgk2t" podStartSLOduration=1.959115741 podStartE2EDuration="1.959115741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 20:43:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 20:43:30.958276363 +0000 UTC m=+2531.112738323" watchObservedRunningTime="2026-07-07 20:43:30.959115741 +0000 UTC m=+2531.113577690" Jul 07 20:43:33.073492 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:33.073449 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4wwvr_4b7e3a9b-4519-44fd-93c2-60850dca6128/dns/0.log" Jul 07 20:43:33.096328 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:33.096295 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4wwvr_4b7e3a9b-4519-44fd-93c2-60850dca6128/kube-rbac-proxy/0.log" Jul 07 20:43:33.223926 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:33.223893 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b8qtm_f3290b11-e223-477b-b593-745224204b1e/dns-node-resolver/0.log" Jul 07 20:43:33.866871 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:33.866837 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jbxhj_f5a8c362-f805-466b-ae7b-ca28c4087e63/node-ca/0.log" Jul 07 20:43:35.034511 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:35.034482 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68f4c445fc-48crl_18992f4a-342c-4bd6-81fb-5e5fc9d951d1/kube-auth-proxy/0.log" Jul 07 20:43:35.995803 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:35.995777 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-87llx_20c78065-0c7e-41b4-91a7-bc885a651f4b/serve-healthcheck-canary/0.log" Jul 07 20:43:36.511720 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:36.511694 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2hcpv_bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a/kube-rbac-proxy/0.log" Jul 07 20:43:36.534695 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:36.534661 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2hcpv_bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a/exporter/0.log" Jul 07 20:43:36.558343 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:36.558302 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2hcpv_bfb1f278-7bd1-4f66-b373-dc5b7ebbfd7a/extractor/0.log" Jul 07 20:43:36.959235 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:36.959203 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ww672/perf-node-gather-daemonset-kgk2t" Jul 07 20:43:39.196063 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:39.196029 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-77476c4d87-twlcf_ce7010ee-0687-4868-9077-7dfdd7a3545e/maas-api/0.log" Jul 07 20:43:39.218775 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:39.218679 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724255-kzznf_8ee5a14b-25d4-40c5-b111-7d5f89653d34/cleanup/0.log" Jul 07 20:43:39.249915 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:39.249876 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724270-qh4mr_0e431ffc-713b-41c8-bd1a-377110182a1c/cleanup/0.log" Jul 07 20:43:39.567790 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:39.567755 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-659495cbcb-pw9cc_510a6e1c-e835-4397-9f70-e1fe37af3f2b/manager/0.log" Jul 07 20:43:39.594147 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:39.594117 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-659495cbcb-pw9cc_510a6e1c-e835-4397-9f70-e1fe37af3f2b/copy-manifests/0.log" Jul 07 20:43:40.470550 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:40.470516 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68f4c445fc-48crl_18992f4a-342c-4bd6-81fb-5e5fc9d951d1/kube-auth-proxy/0.log" Jul 07 20:43:41.111682 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:41.111651 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-699bddd6bc-z9qtq_4ebd5d73-cc74-4811-9393-4c64df8de9e0/manager/0.log" Jul 07 20:43:41.140591 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:41.140510 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-xg82n_4f2f71c8-ba9b-43d4-8b14-6406ac50f2da/openshift-lws-operator/0.log" Jul 07 20:43:41.958772 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:41.958698 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68f4c445fc-48crl_18992f4a-342c-4bd6-81fb-5e5fc9d951d1/kube-auth-proxy/0.log" Jul 07 20:43:47.620673 ip-10-0-133-229 kubenswrapper[2565]: I0707 20:43:47.620633 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-25ms9_1b90c45b-0580-4343-b306-5bcc726279a6/kube-multus/0.log"