Jul 07 17:52:03.529487 ip-10-0-136-192 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 07 17:52:03.529501 ip-10-0-136-192 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 07 17:52:03.529508 ip-10-0-136-192 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 07 17:52:03.529796 ip-10-0-136-192 systemd[1]: Failed to start Kubernetes Kubelet. Jul 07 17:52:13.599755 ip-10-0-136-192 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 07 17:52:13.599777 ip-10-0-136-192 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 38b4242ce4984bc3ac2a3aab1894bd5d -- Jul 07 17:54:10.796263 ip-10-0-136-192 systemd[1]: Starting Kubernetes Kubelet... Jul 07 17:54:11.196030 ip-10-0-136-192 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 17:54:11.196030 ip-10-0-136-192 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 07 17:54:11.196030 ip-10-0-136-192 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 17:54:11.196030 ip-10-0-136-192 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 07 17:54:11.196030 ip-10-0-136-192 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 17:54:11.196724 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.196035 2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 07 17:54:11.200742 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200709 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 17:54:11.200742 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200737 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 17:54:11.200742 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200743 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 17:54:11.200742 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200747 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 17:54:11.200742 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200752 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 17:54:11.200986 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200756 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 17:54:11.200986 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200761 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 17:54:11.200986 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200765 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 17:54:11.200986 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200769 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 17:54:11.200986 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200772 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 17:54:11.200986 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200779 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 17:54:11.200986 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200783 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 17:54:11.200986 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200788 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 17:54:11.200986 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200794 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 17:54:11.200986 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200798 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 17:54:11.200986 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200803 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 17:54:11.200986 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200807 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 17:54:11.200986 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200811 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 17:54:11.200986 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200815 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 17:54:11.200986 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200819 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 17:54:11.200986 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200824 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 17:54:11.200986 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200828 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 17:54:11.200986 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200832 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 17:54:11.200986 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200837 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 17:54:11.200986 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200843 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 17:54:11.201502 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200850 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 17:54:11.201502 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200854 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 17:54:11.201502 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200861 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 17:54:11.201502 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200868 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 17:54:11.201502 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200872 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 17:54:11.201502 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200887 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 17:54:11.201502 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200890 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 17:54:11.201502 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200895 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 17:54:11.201502 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200915 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 17:54:11.201502 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200920 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 17:54:11.201502 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200923 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 17:54:11.201502 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200927 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 17:54:11.201502 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200929 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 17:54:11.201502 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200933 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 17:54:11.201502 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200935 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 17:54:11.201502 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200938 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 17:54:11.201502 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200940 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 17:54:11.201502 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200943 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 17:54:11.201502 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200947 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 17:54:11.201502 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200950 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 17:54:11.201993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200952 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 17:54:11.201993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200955 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 17:54:11.201993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200957 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 17:54:11.201993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200960 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 17:54:11.201993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200962 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 17:54:11.201993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200965 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 17:54:11.201993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200967 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 17:54:11.201993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200970 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 17:54:11.201993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200972 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 17:54:11.201993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200974 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 17:54:11.201993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200977 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 17:54:11.201993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200979 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 17:54:11.201993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200982 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 17:54:11.201993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200984 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 17:54:11.201993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200987 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 17:54:11.201993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200989 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 17:54:11.201993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200993 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 17:54:11.201993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200995 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 17:54:11.201993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.200998 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 17:54:11.202447 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201001 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 17:54:11.202447 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201003 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 17:54:11.202447 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201006 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 17:54:11.202447 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201008 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 17:54:11.202447 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201013 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 17:54:11.202447 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201016 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 17:54:11.202447 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201020 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 17:54:11.202447 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201023 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 17:54:11.202447 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201025 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 17:54:11.202447 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201028 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 17:54:11.202447 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201031 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 17:54:11.202447 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201034 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 17:54:11.202447 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201036 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 17:54:11.202447 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201039 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 17:54:11.202447 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201041 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 17:54:11.202447 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201044 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 17:54:11.202447 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201046 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 17:54:11.202447 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201049 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 17:54:11.202447 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201051 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 17:54:11.202939 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201055 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 17:54:11.202939 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201057 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 17:54:11.202939 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201060 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 17:54:11.202939 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201062 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 17:54:11.202939 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201065 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 17:54:11.202939 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201068 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 17:54:11.202939 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201070 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 07 17:54:11.202939 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201073 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 17:54:11.202939 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201075 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 17:54:11.202939 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201077 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 17:54:11.202939 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201080 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 17:54:11.202939 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201082 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 17:54:11.202939 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201085 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 17:54:11.202939 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201087 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 17:54:11.202939 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201089 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 17:54:11.202939 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201092 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 17:54:11.202939 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201817 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 17:54:11.202939 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201824 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 17:54:11.202939 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201827 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 17:54:11.202939 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201830 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 17:54:11.203425 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201833 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 17:54:11.203425 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201836 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 17:54:11.203425 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201838 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 17:54:11.203425 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201843 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 17:54:11.203425 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201846 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 17:54:11.203425 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201849 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 17:54:11.203425 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201852 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 17:54:11.203425 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201854 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 17:54:11.203425 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201857 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 17:54:11.203425 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201859 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 17:54:11.203425 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201862 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 17:54:11.203425 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201864 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 17:54:11.203425 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201866 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 17:54:11.203425 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201870 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 17:54:11.203425 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201873 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 17:54:11.203425 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201875 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 17:54:11.203425 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201878 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 17:54:11.203425 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201880 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 17:54:11.203425 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201883 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 17:54:11.203425 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201885 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 17:54:11.203972 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201889 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 17:54:11.203972 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201892 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 17:54:11.203972 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201895 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 17:54:11.203972 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201917 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 17:54:11.203972 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201921 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 17:54:11.203972 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201923 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 17:54:11.203972 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201926 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 17:54:11.203972 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201929 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 17:54:11.203972 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201932 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 17:54:11.203972 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201935 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 17:54:11.203972 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201937 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 17:54:11.203972 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201941 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 17:54:11.203972 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201944 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 17:54:11.203972 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201947 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 17:54:11.203972 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201950 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 17:54:11.203972 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201952 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 17:54:11.203972 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201955 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 17:54:11.203972 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201957 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 17:54:11.203972 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201960 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 17:54:11.204528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201962 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 17:54:11.204528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201965 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 17:54:11.204528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201967 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 17:54:11.204528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201970 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 17:54:11.204528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201972 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 17:54:11.204528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201975 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 17:54:11.204528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201977 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 17:54:11.204528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201980 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 17:54:11.204528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201983 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 17:54:11.204528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201985 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 17:54:11.204528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201988 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 17:54:11.204528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201990 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 17:54:11.204528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201993 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 17:54:11.204528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201996 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 17:54:11.204528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.201999 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 17:54:11.204528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202001 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 17:54:11.204528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202004 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 17:54:11.204528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202007 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 17:54:11.204528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202009 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 17:54:11.205002 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202012 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 17:54:11.205002 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202014 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 17:54:11.205002 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202017 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 17:54:11.205002 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202019 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 17:54:11.205002 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202022 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 17:54:11.205002 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202026 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 17:54:11.205002 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202029 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 17:54:11.205002 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202031 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 17:54:11.205002 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202036 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 17:54:11.205002 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202039 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 17:54:11.205002 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202042 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 07 17:54:11.205002 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202044 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 17:54:11.205002 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202047 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 17:54:11.205002 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202049 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 17:54:11.205002 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202051 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 17:54:11.205002 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202055 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 17:54:11.205002 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202058 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 17:54:11.205002 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202060 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 17:54:11.205002 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202062 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 17:54:11.205002 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202065 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 17:54:11.205484 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202068 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 17:54:11.205484 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202071 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 17:54:11.205484 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202073 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 17:54:11.205484 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202076 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 17:54:11.205484 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202078 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 17:54:11.205484 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202080 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 17:54:11.205484 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202083 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 17:54:11.205484 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202086 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 17:54:11.205484 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202089 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 17:54:11.205484 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202091 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 17:54:11.205484 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202094 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 17:54:11.205484 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202096 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 17:54:11.205484 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202099 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 17:54:11.205484 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202101 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 17:54:11.205484 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202103 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 17:54:11.205484 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202106 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 17:54:11.205484 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202108 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 17:54:11.205484 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202529 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 17:54:11.205484 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202535 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 17:54:11.205484 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202538 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 17:54:11.205993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202541 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 17:54:11.205993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202544 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 17:54:11.205993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202546 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 17:54:11.205993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202549 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 17:54:11.205993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202552 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 17:54:11.205993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202554 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 17:54:11.205993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202556 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 17:54:11.205993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202559 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 17:54:11.205993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202561 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 17:54:11.205993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202564 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 17:54:11.205993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202566 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 17:54:11.205993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202569 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 17:54:11.205993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202571 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 17:54:11.205993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202574 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 17:54:11.205993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202577 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 17:54:11.205993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202579 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 17:54:11.205993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202582 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 17:54:11.205993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202584 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 17:54:11.205993 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202588 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 17:54:11.206456 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202591 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 17:54:11.206456 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202594 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 17:54:11.206456 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202597 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 17:54:11.206456 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202599 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 17:54:11.206456 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202602 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 17:54:11.206456 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202605 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 17:54:11.206456 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202607 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 17:54:11.206456 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202610 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 17:54:11.206456 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202612 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 17:54:11.206456 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202615 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 17:54:11.206456 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202620 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 17:54:11.206456 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202624 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 17:54:11.206456 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202627 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 17:54:11.206456 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202631 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 17:54:11.206456 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202633 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 17:54:11.206456 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202635 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 17:54:11.206456 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202638 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 17:54:11.206456 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202641 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 17:54:11.206456 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202643 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 17:54:11.206456 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202645 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 17:54:11.206961 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202648 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 17:54:11.206961 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202650 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 17:54:11.206961 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202653 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 17:54:11.206961 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202655 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 17:54:11.206961 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202658 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 17:54:11.206961 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202660 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 17:54:11.206961 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202662 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 17:54:11.206961 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202665 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 17:54:11.206961 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202668 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 17:54:11.206961 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202672 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 17:54:11.206961 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202674 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 17:54:11.206961 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202676 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 17:54:11.206961 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202679 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 17:54:11.206961 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202681 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 17:54:11.206961 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202684 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 17:54:11.206961 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202686 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 17:54:11.206961 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202688 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 17:54:11.206961 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202691 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 17:54:11.206961 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202693 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 17:54:11.207418 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202695 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 17:54:11.207418 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202698 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 17:54:11.207418 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202701 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 17:54:11.207418 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202704 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 17:54:11.207418 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202706 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 17:54:11.207418 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202708 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 17:54:11.207418 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202711 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 17:54:11.207418 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202714 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 17:54:11.207418 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202716 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 17:54:11.207418 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202718 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 17:54:11.207418 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202721 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 17:54:11.207418 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202723 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 17:54:11.207418 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202725 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 17:54:11.207418 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202728 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 17:54:11.207418 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202730 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 17:54:11.207418 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202733 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 17:54:11.207418 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202735 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 17:54:11.207418 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202737 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 17:54:11.207418 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202740 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 17:54:11.207926 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202742 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 07 17:54:11.207926 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202745 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 17:54:11.207926 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202748 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 17:54:11.207926 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202750 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 17:54:11.207926 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202752 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 17:54:11.207926 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202756 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 17:54:11.207926 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202759 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 17:54:11.207926 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202762 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 17:54:11.207926 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202765 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 17:54:11.207926 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202768 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 17:54:11.207926 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202770 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 17:54:11.207926 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202773 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 17:54:11.207926 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202775 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 17:54:11.207926 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202778 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 17:54:11.207926 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202781 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 17:54:11.207926 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202786 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 17:54:11.207926 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202789 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 17:54:11.207926 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202792 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 17:54:11.207926 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.202794 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 17:54:11.207926 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.202878 2583 flags.go:64] FLAG: --address="0.0.0.0" Jul 07 17:54:11.208430 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.202885 2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 07 17:54:11.208430 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.202893 2583 flags.go:64] FLAG: --anonymous-auth="true" Jul 07 17:54:11.208430 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.202913 2583 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 07 17:54:11.208430 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.202919 2583 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 07 17:54:11.208430 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.202923 2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 07 17:54:11.208430 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.202938 2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 07 17:54:11.208430 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.202943 2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 07 17:54:11.208430 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.202946 2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 07 17:54:11.208430 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.202949 2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 07 17:54:11.208430 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.202953 2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 07 17:54:11.208430 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.202956 2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 07 17:54:11.208430 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.202959 2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 07 17:54:11.208430 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.202962 2583 flags.go:64] FLAG: --cgroup-root="" Jul 07 17:54:11.208430 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.202965 2583 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 07 17:54:11.208430 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.202968 2583 flags.go:64] FLAG: --client-ca-file="" Jul 07 17:54:11.208430 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.202971 2583 flags.go:64] FLAG: --cloud-provider="external" Jul 07 17:54:11.208430 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.202974 2583 flags.go:64] FLAG: --cluster-dns="[]" Jul 07 17:54:11.208430 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.202978 2583 flags.go:64] FLAG: --cluster-domain="" Jul 07 17:54:11.208430 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.202981 2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 07 17:54:11.208430 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.202984 2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 07 17:54:11.208430 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.202987 2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 07 17:54:11.208430 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.202990 2583 flags.go:64] FLAG: --container-log-max-files="5" Jul 07 17:54:11.208430 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.202994 2583 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 07 17:54:11.208430 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.202997 2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 07 17:54:11.209026 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.202999 2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 07 17:54:11.209026 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203003 2583 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 07 17:54:11.209026 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203006 2583 flags.go:64] FLAG: --contention-profiling="false" Jul 07 17:54:11.209026 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203010 2583 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 07 17:54:11.209026 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203014 2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 07 17:54:11.209026 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203017 2583 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 07 17:54:11.209026 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203020 2583 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 07 17:54:11.209026 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203024 2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 07 17:54:11.209026 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203027 2583 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 07 17:54:11.209026 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203030 2583 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 07 17:54:11.209026 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203032 2583 flags.go:64] FLAG: --enable-load-reader="false" Jul 07 17:54:11.209026 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203035 2583 flags.go:64] FLAG: --enable-server="true" Jul 07 17:54:11.209026 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203038 2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 07 17:54:11.209026 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203043 2583 flags.go:64] FLAG: --event-burst="100" Jul 07 17:54:11.209026 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203046 2583 flags.go:64] FLAG: --event-qps="50" Jul 07 17:54:11.209026 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203048 2583 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 07 17:54:11.209026 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203051 2583 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 07 17:54:11.209026 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203054 2583 flags.go:64] FLAG: --eviction-hard="" Jul 07 17:54:11.209026 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203058 2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 07 17:54:11.209026 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203061 2583 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 07 17:54:11.209026 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203063 2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 07 17:54:11.209026 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203066 2583 flags.go:64] FLAG: --eviction-soft="" Jul 07 17:54:11.209026 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203069 2583 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 07 17:54:11.209026 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203072 2583 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 07 17:54:11.209026 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203075 2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 07 17:54:11.209700 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203078 2583 flags.go:64] FLAG: --experimental-mounter-path="" Jul 07 17:54:11.209700 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203082 2583 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 07 17:54:11.209700 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203084 2583 flags.go:64] FLAG: --fail-swap-on="true" Jul 07 17:54:11.209700 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203087 2583 flags.go:64] FLAG: --feature-gates="" Jul 07 17:54:11.209700 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203091 2583 flags.go:64] FLAG: --file-check-frequency="20s" Jul 07 17:54:11.209700 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203094 2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 07 17:54:11.209700 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203097 2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 07 17:54:11.209700 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203100 2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 07 17:54:11.209700 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203102 2583 flags.go:64] FLAG: --healthz-port="10248" Jul 07 17:54:11.209700 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203106 2583 flags.go:64] FLAG: --help="false" Jul 07 17:54:11.209700 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203109 2583 flags.go:64] FLAG: --hostname-override="ip-10-0-136-192.ec2.internal" Jul 07 17:54:11.209700 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203114 2583 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 07 17:54:11.209700 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203117 2583 flags.go:64] FLAG: --http-check-frequency="20s" Jul 07 17:54:11.209700 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203119 2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 07 17:54:11.209700 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203123 2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 07 17:54:11.209700 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203126 2583 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 07 17:54:11.209700 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203129 2583 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 07 17:54:11.209700 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203132 2583 flags.go:64] FLAG: --image-service-endpoint="" Jul 07 17:54:11.209700 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203134 2583 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 07 17:54:11.209700 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203137 2583 flags.go:64] FLAG: --kube-api-burst="100" Jul 07 17:54:11.209700 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203140 2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 07 17:54:11.209700 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203143 2583 flags.go:64] FLAG: --kube-api-qps="50" Jul 07 17:54:11.209700 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203145 2583 flags.go:64] FLAG: --kube-reserved="" Jul 07 17:54:11.209700 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203148 2583 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 07 17:54:11.210534 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203151 2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 07 17:54:11.210534 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203153 2583 flags.go:64] FLAG: --kubelet-cgroups="" Jul 07 17:54:11.210534 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203156 2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 07 17:54:11.210534 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203159 2583 flags.go:64] FLAG: --lock-file="" Jul 07 17:54:11.210534 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203162 2583 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 07 17:54:11.210534 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203165 2583 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 07 17:54:11.210534 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203167 2583 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 07 17:54:11.210534 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203173 2583 flags.go:64] FLAG: --log-json-split-stream="false" Jul 07 17:54:11.210534 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203176 2583 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 07 17:54:11.210534 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203179 2583 flags.go:64] FLAG: --log-text-split-stream="false" Jul 07 17:54:11.210534 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203182 2583 flags.go:64] FLAG: --logging-format="text" Jul 07 17:54:11.210534 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203184 2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 07 17:54:11.210534 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203187 2583 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 07 17:54:11.210534 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203190 2583 flags.go:64] FLAG: --manifest-url="" Jul 07 17:54:11.210534 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203192 2583 flags.go:64] FLAG: --manifest-url-header="" Jul 07 17:54:11.210534 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203197 2583 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 07 17:54:11.210534 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203200 2583 flags.go:64] FLAG: --max-open-files="1000000" Jul 07 17:54:11.210534 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203204 2583 flags.go:64] FLAG: --max-pods="110" Jul 07 17:54:11.210534 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203208 2583 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 07 17:54:11.210534 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203212 2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 07 17:54:11.210534 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203215 2583 flags.go:64] FLAG: --memory-manager-policy="None" Jul 07 17:54:11.210534 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203218 2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 07 17:54:11.210534 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203221 2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 07 17:54:11.210534 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203223 2583 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 07 17:54:11.210534 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203226 2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 07 17:54:11.211616 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203233 2583 flags.go:64] FLAG: --node-status-max-images="50" Jul 07 17:54:11.211616 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203236 2583 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 07 17:54:11.211616 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203239 2583 flags.go:64] FLAG: --oom-score-adj="-999" Jul 07 17:54:11.211616 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203242 2583 flags.go:64] FLAG: --pod-cidr="" Jul 07 17:54:11.211616 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203245 2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 07 17:54:11.211616 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203250 2583 flags.go:64] FLAG: --pod-manifest-path="" Jul 07 17:54:11.211616 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203253 2583 flags.go:64] FLAG: --pod-max-pids="-1" Jul 07 17:54:11.211616 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203256 2583 flags.go:64] FLAG: --pods-per-core="0" Jul 07 17:54:11.211616 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203259 2583 flags.go:64] FLAG: --port="10250" Jul 07 17:54:11.211616 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203261 2583 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 07 17:54:11.211616 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203264 2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07f070c67d6fc1add" Jul 07 17:54:11.211616 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203267 2583 flags.go:64] FLAG: --qos-reserved="" Jul 07 17:54:11.211616 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203270 2583 flags.go:64] FLAG: --read-only-port="10255" Jul 07 17:54:11.211616 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203273 2583 flags.go:64] FLAG: --register-node="true" Jul 07 17:54:11.211616 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203276 2583 flags.go:64] FLAG: --register-with-taints="" Jul 07 17:54:11.211616 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203279 2583 flags.go:64] FLAG: --registry-burst="10" Jul 07 17:54:11.211616 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203283 2583 flags.go:64] FLAG: --registry-qps="5" Jul 07 17:54:11.211616 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203285 2583 flags.go:64] FLAG: --reserved-cpus="" Jul 07 17:54:11.211616 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203288 2583 flags.go:64] FLAG: --reserved-memory="" Jul 07 17:54:11.211616 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203292 2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 07 17:54:11.211616 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203294 2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 07 17:54:11.211616 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203297 2583 flags.go:64] FLAG: --rotate-certificates="false" Jul 07 17:54:11.211616 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203300 2583 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 07 17:54:11.211616 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203303 2583 flags.go:64] FLAG: --runonce="false" Jul 07 17:54:11.211616 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203305 2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 07 17:54:11.212696 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203308 2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 07 17:54:11.212696 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203313 2583 flags.go:64] FLAG: --seccomp-default="false" Jul 07 17:54:11.212696 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203316 2583 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 07 17:54:11.212696 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203319 2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 07 17:54:11.212696 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203322 2583 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 07 17:54:11.212696 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203325 2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 07 17:54:11.212696 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203327 2583 flags.go:64] FLAG: --storage-driver-password="root" Jul 07 17:54:11.212696 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203330 2583 flags.go:64] FLAG: --storage-driver-secure="false" Jul 07 17:54:11.212696 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203333 2583 flags.go:64] FLAG: --storage-driver-table="stats" Jul 07 17:54:11.212696 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203336 2583 flags.go:64] FLAG: --storage-driver-user="root" Jul 07 17:54:11.212696 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203338 2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 07 17:54:11.212696 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203341 2583 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 07 17:54:11.212696 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203344 2583 flags.go:64] FLAG: --system-cgroups="" Jul 07 17:54:11.212696 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203347 2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 07 17:54:11.212696 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203353 2583 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 07 17:54:11.212696 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203355 2583 flags.go:64] FLAG: --tls-cert-file="" Jul 07 17:54:11.212696 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203358 2583 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 07 17:54:11.212696 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203362 2583 flags.go:64] FLAG: --tls-min-version="" Jul 07 17:54:11.212696 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203365 2583 flags.go:64] FLAG: --tls-private-key-file="" Jul 07 17:54:11.212696 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203367 2583 flags.go:64] FLAG: --topology-manager-policy="none" Jul 07 17:54:11.212696 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203370 2583 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 07 17:54:11.212696 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203373 2583 flags.go:64] FLAG: --topology-manager-scope="container" Jul 07 17:54:11.212696 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203375 2583 flags.go:64] FLAG: --v="2" Jul 07 17:54:11.212696 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203379 2583 flags.go:64] FLAG: --version="false" Jul 07 17:54:11.212696 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203384 2583 flags.go:64] FLAG: --vmodule="" Jul 07 17:54:11.213769 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203387 2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 07 17:54:11.213769 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.203390 2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 07 17:54:11.213769 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203514 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 17:54:11.213769 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203518 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 17:54:11.213769 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203521 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 17:54:11.213769 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203524 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 17:54:11.213769 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203526 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 17:54:11.213769 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203529 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 17:54:11.213769 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203536 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 17:54:11.213769 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203538 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 17:54:11.213769 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203541 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 17:54:11.213769 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203543 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 17:54:11.213769 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203545 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 17:54:11.213769 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203548 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 17:54:11.213769 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203551 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 17:54:11.213769 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203553 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 17:54:11.213769 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203556 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 17:54:11.213769 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203559 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 17:54:11.213769 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203561 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 17:54:11.213769 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203564 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 17:54:11.214615 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203567 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 17:54:11.214615 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203569 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 17:54:11.214615 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203571 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 17:54:11.214615 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203574 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 17:54:11.214615 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203577 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 17:54:11.214615 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203579 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 17:54:11.214615 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203582 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 17:54:11.214615 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203584 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 17:54:11.214615 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203587 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 17:54:11.214615 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203590 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 17:54:11.214615 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203592 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 17:54:11.214615 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203594 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 17:54:11.214615 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203597 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 17:54:11.214615 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203600 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 17:54:11.214615 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203603 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 17:54:11.214615 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203605 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 17:54:11.214615 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203608 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 17:54:11.214615 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203610 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 17:54:11.214615 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203612 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 17:54:11.214615 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203615 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 17:54:11.215399 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203620 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 17:54:11.215399 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203622 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 17:54:11.215399 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203625 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 17:54:11.215399 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203627 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 17:54:11.215399 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203630 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 17:54:11.215399 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203633 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 17:54:11.215399 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203635 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 17:54:11.215399 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203637 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 17:54:11.215399 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203640 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 17:54:11.215399 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203642 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 17:54:11.215399 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203645 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 17:54:11.215399 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203647 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 17:54:11.215399 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203650 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 17:54:11.215399 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203652 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 17:54:11.215399 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203655 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 17:54:11.215399 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203657 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 17:54:11.215399 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203660 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 17:54:11.215399 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203663 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 17:54:11.215399 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203665 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 17:54:11.215399 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203668 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 17:54:11.216254 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203670 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 17:54:11.216254 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203673 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 17:54:11.216254 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203675 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 17:54:11.216254 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203677 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 17:54:11.216254 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203680 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 17:54:11.216254 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203683 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 17:54:11.216254 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203686 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 17:54:11.216254 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203688 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 17:54:11.216254 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203690 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 17:54:11.216254 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203693 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 17:54:11.216254 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203695 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 17:54:11.216254 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203697 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 17:54:11.216254 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203703 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 17:54:11.216254 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203707 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 17:54:11.216254 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203711 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 17:54:11.216254 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203714 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 17:54:11.216254 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203716 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 07 17:54:11.216254 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203719 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 17:54:11.216254 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203722 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 17:54:11.216853 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203726 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 17:54:11.216853 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203729 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 17:54:11.216853 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203732 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 17:54:11.216853 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203735 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 17:54:11.216853 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203738 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 17:54:11.216853 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203740 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 17:54:11.216853 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203743 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 17:54:11.216853 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203745 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 17:54:11.216853 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203748 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 17:54:11.216853 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203750 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 17:54:11.216853 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203753 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 17:54:11.216853 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203755 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 17:54:11.216853 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203758 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 17:54:11.216853 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203760 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 17:54:11.216853 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203763 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 17:54:11.216853 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203765 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 17:54:11.216853 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203768 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 17:54:11.216853 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203770 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 17:54:11.216853 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203773 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 17:54:11.216853 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203776 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 17:54:11.217615 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203790 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 17:54:11.217615 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.203794 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 17:54:11.217615 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.204492 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 17:54:11.217615 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.212819 2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 07 17:54:11.217615 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.212851 2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 07 17:54:11.217615 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.212879 2583 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 07 17:54:11.217615 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.212889 2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 07 17:54:11.217615 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213006 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 17:54:11.217615 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213015 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 17:54:11.217615 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213020 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 17:54:11.217615 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213024 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 17:54:11.217615 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213029 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 17:54:11.217615 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213033 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 17:54:11.217615 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213038 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 17:54:11.218048 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213043 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 17:54:11.218048 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213047 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 17:54:11.218048 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213051 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 17:54:11.218048 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213055 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 17:54:11.218048 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213060 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 17:54:11.218048 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213065 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 17:54:11.218048 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213069 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 17:54:11.218048 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213073 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 17:54:11.218048 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213077 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 17:54:11.218048 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213081 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 17:54:11.218048 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213085 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 17:54:11.218048 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213089 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 17:54:11.218048 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213093 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 17:54:11.218048 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213097 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 17:54:11.218048 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213101 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 17:54:11.218048 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213105 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 17:54:11.218048 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213109 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 17:54:11.218048 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213113 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 17:54:11.218048 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213117 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 17:54:11.218048 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213121 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 17:54:11.218548 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213125 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 17:54:11.218548 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213130 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 17:54:11.218548 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213143 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 17:54:11.218548 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213148 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 17:54:11.218548 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213151 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 17:54:11.218548 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213156 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 17:54:11.218548 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213160 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 17:54:11.218548 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213164 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 17:54:11.218548 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213168 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 17:54:11.218548 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213172 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 17:54:11.218548 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213176 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 17:54:11.218548 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213181 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 17:54:11.218548 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213185 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 17:54:11.218548 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213189 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 17:54:11.218548 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213193 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 17:54:11.218548 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213198 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 17:54:11.218548 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213204 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 17:54:11.218548 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213211 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 17:54:11.218548 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213218 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 17:54:11.219113 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213224 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 17:54:11.219113 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213230 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 17:54:11.219113 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213235 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 17:54:11.219113 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213240 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 17:54:11.219113 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213245 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 17:54:11.219113 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213250 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 17:54:11.219113 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213254 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 17:54:11.219113 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213259 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 17:54:11.219113 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213265 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 17:54:11.219113 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213269 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 17:54:11.219113 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213274 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 17:54:11.219113 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213278 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 17:54:11.219113 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213282 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 17:54:11.219113 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213286 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 17:54:11.219113 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213291 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 17:54:11.219113 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213296 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 17:54:11.219113 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213310 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 17:54:11.219113 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213315 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 17:54:11.219113 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213320 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 17:54:11.219649 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213324 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 17:54:11.219649 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213329 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 17:54:11.219649 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213333 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 17:54:11.219649 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213338 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 17:54:11.219649 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213342 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 07 17:54:11.219649 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213346 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 17:54:11.219649 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213350 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 17:54:11.219649 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213354 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 17:54:11.219649 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213358 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 17:54:11.219649 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213363 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 17:54:11.219649 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213367 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 17:54:11.219649 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213371 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 17:54:11.219649 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213375 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 17:54:11.219649 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213379 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 17:54:11.219649 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213384 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 17:54:11.219649 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213388 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 17:54:11.219649 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213392 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 17:54:11.219649 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213396 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 17:54:11.219649 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213401 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 17:54:11.219649 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213405 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 17:54:11.220199 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213409 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 17:54:11.220199 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213413 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 17:54:11.220199 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213417 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 17:54:11.220199 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213421 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 17:54:11.220199 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213426 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 17:54:11.220199 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213429 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 17:54:11.220199 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213433 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 17:54:11.220199 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213437 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 17:54:11.220199 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213442 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 17:54:11.220199 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213446 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 17:54:11.220199 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213458 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 17:54:11.220199 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213463 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 17:54:11.220199 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213467 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 17:54:11.220199 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213471 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 17:54:11.220199 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.213479 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 17:54:11.220607 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213698 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 17:54:11.220607 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213706 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 17:54:11.220607 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213711 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 17:54:11.220607 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213716 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 17:54:11.220607 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213720 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 17:54:11.220607 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213724 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 17:54:11.220607 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213728 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 17:54:11.220607 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213732 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 17:54:11.220607 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213736 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 17:54:11.220607 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213740 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 17:54:11.220607 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213744 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 17:54:11.220607 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213748 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 17:54:11.220607 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213752 2583 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 17:54:11.220607 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213757 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 17:54:11.220607 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213761 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 17:54:11.220607 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213765 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 17:54:11.220607 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213769 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 17:54:11.220607 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213772 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 17:54:11.220607 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213776 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 17:54:11.220607 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213779 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 17:54:11.221347 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213783 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 17:54:11.221347 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213787 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 17:54:11.221347 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213790 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 17:54:11.221347 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213795 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 17:54:11.221347 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213798 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 17:54:11.221347 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213802 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 17:54:11.221347 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213807 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 17:54:11.221347 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213811 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 17:54:11.221347 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213826 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 17:54:11.221347 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213831 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 17:54:11.221347 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213835 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 17:54:11.221347 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213839 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 17:54:11.221347 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213843 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 17:54:11.221347 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213849 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 17:54:11.221347 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213855 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 17:54:11.221347 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213859 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 17:54:11.221347 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213864 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 17:54:11.221347 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213868 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 17:54:11.221347 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213873 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 17:54:11.222175 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213877 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 17:54:11.222175 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213882 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 17:54:11.222175 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213886 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 17:54:11.222175 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213890 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 17:54:11.222175 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213894 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 17:54:11.222175 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213921 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 17:54:11.222175 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213926 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 17:54:11.222175 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213931 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 17:54:11.222175 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213935 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 17:54:11.222175 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213939 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 17:54:11.222175 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213943 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 17:54:11.222175 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213947 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 17:54:11.222175 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213951 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 17:54:11.222175 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213955 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 17:54:11.222175 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213959 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 17:54:11.222175 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213963 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 17:54:11.222175 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213967 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 17:54:11.222175 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213971 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 17:54:11.222175 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213975 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 17:54:11.222175 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213979 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 17:54:11.222912 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213983 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 17:54:11.222912 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.213988 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 17:54:11.222912 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214000 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 17:54:11.222912 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214005 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 17:54:11.222912 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214009 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 17:54:11.222912 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214013 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 17:54:11.222912 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214017 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 17:54:11.222912 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214021 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 17:54:11.222912 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214025 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 17:54:11.222912 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214029 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 17:54:11.222912 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214033 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 17:54:11.222912 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214037 2583 feature_gate.go:328] unrecognized feature gate: Example Jul 07 17:54:11.222912 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214041 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 17:54:11.222912 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214045 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 17:54:11.222912 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214049 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 17:54:11.222912 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214053 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 17:54:11.222912 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214058 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 17:54:11.222912 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214062 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 17:54:11.222912 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214066 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 17:54:11.222912 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214070 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 17:54:11.223528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214074 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 17:54:11.223528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214078 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 17:54:11.223528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214083 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 17:54:11.223528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214087 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 17:54:11.223528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214091 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 17:54:11.223528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214095 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 17:54:11.223528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214099 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 17:54:11.223528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214103 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 17:54:11.223528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214107 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 17:54:11.223528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214113 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 17:54:11.223528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214119 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 17:54:11.223528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214123 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 17:54:11.223528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214128 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 17:54:11.223528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214132 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 17:54:11.223528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214136 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 17:54:11.223528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214148 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 17:54:11.223528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214153 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 17:54:11.223528 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214157 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 17:54:11.223999 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214164 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 17:54:11.223999 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:11.214168 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 17:54:11.223999 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.214177 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 17:54:11.223999 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.214392 2583 server.go:962] "Client rotation is on, will bootstrap in background" Jul 07 17:54:11.223999 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.217148 2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 07 17:54:11.223999 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.218604 2583 server.go:1019] "Starting client certificate rotation" Jul 07 17:54:11.223999 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.218714 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 17:54:11.223999 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.219509 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 17:54:11.223999 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.221719 2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 17:54:11.227644 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.227616 2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 17:54:11.238782 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.238756 2583 log.go:25] "Validated CRI v1 runtime API" Jul 07 17:54:11.240960 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.240944 2583 log.go:25] "Validated CRI v1 image API" Jul 07 17:54:11.241695 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.241680 2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 07 17:54:11.244519 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.244496 2583 fs.go:135] Filesystem UUIDs: map[2ec5b289-d762-4b94-b497-99dd9a279daf:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 eae7d084-659a-4ac1-91fa-d5c50d4b9044:/dev/nvme0n1p4] Jul 07 17:54:11.244605 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.244518 2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 07 17:54:11.248867 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.248846 2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 17:54:11.249847 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.249725 2583 manager.go:216] Machine: {Timestamp:2026-07-07 17:54:11.248353987 +0000 UTC m=+0.330957683 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3116512 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bb81fa3e4c790d597f40850e6f4a0 SystemUUID:ec2bb81f-a3e4-c790-d597-f40850e6f4a0 BootID:38b4242c-e498-4bc3-ac2a-3aab1894bd5d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:00:36:a0:c0:21 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:00:36:a0:c0:21 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:b1:23:91:5e:33 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 07 17:54:11.249847 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.249843 2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 07 17:54:11.249990 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.249961 2583 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 07 17:54:11.253332 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.253295 2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 07 17:54:11.253480 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.253335 2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-192.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 07 17:54:11.254464 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.254452 2583 topology_manager.go:138] "Creating topology manager with none policy" Jul 07 17:54:11.254497 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.254467 2583 container_manager_linux.go:309] "Creating device plugin manager" Jul 07 17:54:11.254497 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.254475 2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 17:54:11.255169 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.255157 2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 17:54:11.256450 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.256437 2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 07 17:54:11.259180 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.259167 2583 state_mem.go:36] "Initialized new in-memory state store" Jul 07 17:54:11.259314 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.259304 2583 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 07 17:54:11.259348 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.259337 2583 kubelet.go:486] "Attempting to sync node with API server" Jul 07 17:54:11.259404 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.259374 2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 07 17:54:11.259404 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.259398 2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 07 17:54:11.259470 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.259410 2583 kubelet.go:398] "Adding apiserver pod source" Jul 07 17:54:11.259470 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.259419 2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 07 17:54:11.259790 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.259778 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 17:54:11.259847 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.259795 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 17:54:11.262104 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.262079 2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 07 17:54:11.264023 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.264003 2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 07 17:54:11.264115 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.264044 2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 07 17:54:11.265215 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.265201 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 07 17:54:11.265681 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.265221 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 07 17:54:11.265681 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.265231 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 07 17:54:11.265681 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.265241 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 07 17:54:11.265681 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.265253 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 07 17:54:11.265681 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.265261 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 07 17:54:11.265681 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.265272 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 07 17:54:11.265681 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.265280 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 07 17:54:11.265681 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.265289 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 07 17:54:11.265681 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.265492 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 07 17:54:11.265681 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.265624 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 07 17:54:11.265681 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.265639 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 07 17:54:11.267066 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.267051 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 07 17:54:11.267066 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.267068 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 07 17:54:11.270645 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.270627 2583 server.go:1268] "Started kubelet" Jul 07 17:54:11.270770 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.270723 2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 07 17:54:11.270861 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.270705 2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 07 17:54:11.271981 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.271943 2583 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 07 17:54:11.272658 ip-10-0-136-192 systemd[1]: Started Kubernetes Kubelet. Jul 07 17:54:11.273611 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:11.273509 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 07 17:54:11.273781 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.273760 2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 07 17:54:11.273859 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:11.273815 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-192.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 07 17:54:11.275816 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.275789 2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-192.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 07 17:54:11.276391 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.276376 2583 server.go:310] "Adding debug handlers to kubelet server" Jul 07 17:54:11.279318 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:11.279294 2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 07 17:54:11.281566 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:11.280520 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-192.ec2.internal.18c013764dc5ae99 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-192.ec2.internal,UID:ip-10-0-136-192.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-192.ec2.internal,},FirstTimestamp:2026-07-07 17:54:11.270586009 +0000 UTC m=+0.353189704,LastTimestamp:2026-07-07 17:54:11.270586009 +0000 UTC m=+0.353189704,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-192.ec2.internal,}" Jul 07 17:54:11.283089 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.283071 2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 07 17:54:11.283089 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.283084 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 07 17:54:11.283235 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:11.283204 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-192.ec2.internal\" not found" Jul 07 17:54:11.283850 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.283830 2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 07 17:54:11.283983 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.283835 2583 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 07 17:54:11.283983 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.283866 2583 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 07 17:54:11.283983 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.283976 2583 reconstruct.go:97] "Volume reconstruction finished" Jul 07 17:54:11.284115 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.283990 2583 reconciler.go:29] "Reconciler: start to sync state" Jul 07 17:54:11.284115 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.284002 2583 factory.go:55] Registering systemd factory Jul 07 17:54:11.284115 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.284019 2583 factory.go:223] Registration of the systemd container factory successfully Jul 07 17:54:11.284268 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.284252 2583 factory.go:153] Registering CRI-O factory Jul 07 17:54:11.284268 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.284269 2583 factory.go:223] Registration of the crio container factory successfully Jul 07 17:54:11.284420 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.284321 2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 07 17:54:11.284420 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.284347 2583 factory.go:103] Registering Raw factory Jul 07 17:54:11.284420 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.284364 2583 manager.go:1196] Started watching for new ooms in manager Jul 07 17:54:11.284896 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.284885 2583 manager.go:349] Starting recovery of all containers Jul 07 17:54:11.290306 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:11.290274 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 07 17:54:11.290431 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:11.290297 2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-192.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 07 17:54:11.292028 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.291995 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 07 17:54:11.292140 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.292058 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6m8bw" Jul 07 17:54:11.295504 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.295293 2583 manager.go:354] Recovery completed Jul 07 17:54:11.298104 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:11.298082 2583 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 07 17:54:11.303159 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.303120 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:54:11.303615 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.303598 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6m8bw" Jul 07 17:54:11.305249 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.305229 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-192.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:54:11.305346 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.305266 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-192.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:54:11.305346 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.305281 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-192.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:54:11.306088 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.306066 2583 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 07 17:54:11.306088 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.306082 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:54:11.306226 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.306086 2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 07 17:54:11.306226 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.306147 2583 state_mem.go:36] "Initialized new in-memory state store" Jul 07 17:54:11.306941 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.306920 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-192.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:54:11.307066 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.306962 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-192.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:54:11.307066 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.306979 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-192.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:54:11.307166 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.307073 2583 policy_none.go:49] "None policy: Start" Jul 07 17:54:11.307166 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.307089 2583 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 07 17:54:11.307166 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.307103 2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 07 17:54:11.307474 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:11.307407 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-192.ec2.internal.18c013764fd6940e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-192.ec2.internal,UID:ip-10-0-136-192.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-192.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-192.ec2.internal,},FirstTimestamp:2026-07-07 17:54:11.305247758 +0000 UTC m=+0.387851467,LastTimestamp:2026-07-07 17:54:11.305247758 +0000 UTC m=+0.387851467,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-192.ec2.internal,}" Jul 07 17:54:11.308041 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.308029 2583 policy_none.go:47] "Start" Jul 07 17:54:11.344648 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.344626 2583 manager.go:337] "Starting Device Plugin manager" Jul 07 17:54:11.344786 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:11.344673 2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 07 17:54:11.344786 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.344686 2583 server.go:87] "Starting device plugin registration server" Jul 07 17:54:11.345052 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.345037 2583 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 07 17:54:11.345126 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.345053 2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 07 17:54:11.345169 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.345121 2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 07 17:54:11.345223 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.345213 2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 07 17:54:11.345223 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.345222 2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 07 17:54:11.345820 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:11.345793 2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 07 17:54:11.345931 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:11.345847 2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-192.ec2.internal\" not found" Jul 07 17:54:11.422494 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.422464 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 07 17:54:11.423486 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.422515 2583 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 07 17:54:11.423486 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.422552 2583 kubelet.go:2448] "Starting kubelet main sync loop" Jul 07 17:54:11.423486 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:11.422598 2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 07 17:54:11.425083 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.425060 2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 17:54:11.446238 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.446137 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:54:11.447243 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.447223 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-192.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:54:11.447346 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.447256 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-192.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:54:11.447346 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.447266 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-192.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:54:11.447346 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.447294 2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-192.ec2.internal" Jul 07 17:54:11.454107 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.454086 2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-192.ec2.internal" Jul 07 17:54:11.454167 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:11.454116 2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-192.ec2.internal\": node \"ip-10-0-136-192.ec2.internal\" not found" Jul 07 17:54:11.484245 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:11.484207 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-192.ec2.internal\" not found" Jul 07 17:54:11.523576 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.523525 2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-192.ec2.internal"] Jul 07 17:54:11.523765 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.523635 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:54:11.525287 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.525250 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-192.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:54:11.525287 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.525290 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-192.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:54:11.525455 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.525304 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-192.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:54:11.526545 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.526529 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:54:11.526694 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.526680 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.ec2.internal" Jul 07 17:54:11.526744 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.526714 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:54:11.527327 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.527299 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-192.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:54:11.527327 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.527327 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-192.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:54:11.527433 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.527341 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-192.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:54:11.527433 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.527375 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-192.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:54:11.527433 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.527402 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-192.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:54:11.527433 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.527415 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-192.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:54:11.528418 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.528406 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-192.ec2.internal" Jul 07 17:54:11.528462 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.528438 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 17:54:11.529152 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.529123 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-192.ec2.internal" event="NodeHasSufficientMemory" Jul 07 17:54:11.529218 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.529163 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-192.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 17:54:11.529218 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.529179 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-192.ec2.internal" event="NodeHasSufficientPID" Jul 07 17:54:11.553788 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:11.553759 2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-192.ec2.internal\" not found" node="ip-10-0-136-192.ec2.internal" Jul 07 17:54:11.558469 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:11.558450 2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-192.ec2.internal\" not found" node="ip-10-0-136-192.ec2.internal" Jul 07 17:54:11.584421 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:11.584388 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-192.ec2.internal\" not found" Jul 07 17:54:11.684984 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:11.684948 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-192.ec2.internal\" not found" Jul 07 17:54:11.685176 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.685050 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf831b3419f5fd8b4190fd80bde7e0ae-config\") pod \"kube-apiserver-proxy-ip-10-0-136-192.ec2.internal\" (UID: \"cf831b3419f5fd8b4190fd80bde7e0ae\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-192.ec2.internal" Jul 07 17:54:11.685176 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.685085 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b6ba5b519a9e1c4d8b4f40d858bd5c2f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-192.ec2.internal\" (UID: \"b6ba5b519a9e1c4d8b4f40d858bd5c2f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.ec2.internal" Jul 07 17:54:11.685176 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.685111 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6ba5b519a9e1c4d8b4f40d858bd5c2f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-192.ec2.internal\" (UID: \"b6ba5b519a9e1c4d8b4f40d858bd5c2f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.ec2.internal" Jul 07 17:54:11.785279 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:11.785190 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-192.ec2.internal\" not found" Jul 07 17:54:11.785279 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.785255 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b6ba5b519a9e1c4d8b4f40d858bd5c2f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-192.ec2.internal\" (UID: \"b6ba5b519a9e1c4d8b4f40d858bd5c2f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.ec2.internal" Jul 07 17:54:11.785456 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.785284 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6ba5b519a9e1c4d8b4f40d858bd5c2f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-192.ec2.internal\" (UID: \"b6ba5b519a9e1c4d8b4f40d858bd5c2f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.ec2.internal" Jul 07 17:54:11.785456 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.785304 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf831b3419f5fd8b4190fd80bde7e0ae-config\") pod \"kube-apiserver-proxy-ip-10-0-136-192.ec2.internal\" (UID: \"cf831b3419f5fd8b4190fd80bde7e0ae\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-192.ec2.internal" Jul 07 17:54:11.785456 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.785353 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf831b3419f5fd8b4190fd80bde7e0ae-config\") pod \"kube-apiserver-proxy-ip-10-0-136-192.ec2.internal\" (UID: \"cf831b3419f5fd8b4190fd80bde7e0ae\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-192.ec2.internal" Jul 07 17:54:11.785456 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.785366 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b6ba5b519a9e1c4d8b4f40d858bd5c2f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-192.ec2.internal\" (UID: \"b6ba5b519a9e1c4d8b4f40d858bd5c2f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.ec2.internal" Jul 07 17:54:11.785456 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.785376 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6ba5b519a9e1c4d8b4f40d858bd5c2f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-192.ec2.internal\" (UID: \"b6ba5b519a9e1c4d8b4f40d858bd5c2f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.ec2.internal" Jul 07 17:54:11.856363 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.856321 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.ec2.internal" Jul 07 17:54:11.861130 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:11.861111 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-192.ec2.internal" Jul 07 17:54:11.885950 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:11.885914 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-192.ec2.internal\" not found" Jul 07 17:54:11.987215 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:11.987167 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-192.ec2.internal\" not found" Jul 07 17:54:12.088440 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:12.088353 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-192.ec2.internal\" not found" Jul 07 17:54:12.189156 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:12.189127 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-192.ec2.internal\" not found" Jul 07 17:54:12.219619 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:12.219590 2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 07 17:54:12.220212 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:12.219760 2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 17:54:12.289729 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:12.289689 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-192.ec2.internal\" not found" Jul 07 17:54:12.305961 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:12.305899 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 17:49:11 +0000 UTC" deadline="2028-04-11 08:20:27.692031496 +0000 UTC" Jul 07 17:54:12.305961 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:12.305951 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15446h26m15.386084032s" Jul 07 17:54:12.390167 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:12.390072 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-192.ec2.internal\" not found" Jul 07 17:54:12.412365 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:12.412331 2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 17:54:12.420012 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:12.419972 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6ba5b519a9e1c4d8b4f40d858bd5c2f.slice/crio-a0eb57195cdcfb1a7d0279c37cd536b77d6b1189f4feb0fdb3ffbbb7c87226aa WatchSource:0}: Error finding container a0eb57195cdcfb1a7d0279c37cd536b77d6b1189f4feb0fdb3ffbbb7c87226aa: Status 404 returned error can't find the container with id a0eb57195cdcfb1a7d0279c37cd536b77d6b1189f4feb0fdb3ffbbb7c87226aa Jul 07 17:54:12.420249 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:12.420227 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf831b3419f5fd8b4190fd80bde7e0ae.slice/crio-e0153d116301deb9572d2464d3483136c72c8ae8d74b0300e95a045a223f7c3f WatchSource:0}: Error finding container e0153d116301deb9572d2464d3483136c72c8ae8d74b0300e95a045a223f7c3f: Status 404 returned error can't find the container with id e0153d116301deb9572d2464d3483136c72c8ae8d74b0300e95a045a223f7c3f Jul 07 17:54:12.424258 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:12.424232 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 17:54:12.425614 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:12.425572 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-192.ec2.internal" event={"ID":"cf831b3419f5fd8b4190fd80bde7e0ae","Type":"ContainerStarted","Data":"e0153d116301deb9572d2464d3483136c72c8ae8d74b0300e95a045a223f7c3f"} Jul 07 17:54:12.426765 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:12.426741 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.ec2.internal" event={"ID":"b6ba5b519a9e1c4d8b4f40d858bd5c2f","Type":"ContainerStarted","Data":"a0eb57195cdcfb1a7d0279c37cd536b77d6b1189f4feb0fdb3ffbbb7c87226aa"} Jul 07 17:54:12.490597 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:12.490552 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-192.ec2.internal\" not found" Jul 07 17:54:12.590755 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:12.590713 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-192.ec2.internal\" not found" Jul 07 17:54:12.673689 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:12.673609 2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 17:54:12.683411 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:12.683376 2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.ec2.internal" Jul 07 17:54:12.695574 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:12.695538 2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 17:54:12.696694 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:12.696673 2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-192.ec2.internal" Jul 07 17:54:12.707887 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:12.707856 2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 17:54:12.799792 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:12.799756 2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 17:54:13.260731 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.260695 2583 apiserver.go:52] "Watching apiserver" Jul 07 17:54:13.268656 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.268608 2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 07 17:54:13.269149 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.269113 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-t2g29","openshift-dns/node-resolver-p4mnt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.ec2.internal","openshift-network-operator/iptables-alerter-lbwbm","openshift-ovn-kubernetes/ovnkube-node-zjzpw","kube-system/global-pull-secret-syncer-xqq9h","kube-system/konnectivity-agent-6zwkl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wfrk","openshift-image-registry/node-ca-dm6hm","openshift-multus/multus-additional-cni-plugins-49qq2","openshift-multus/multus-z68qr","openshift-multus/network-metrics-daemon-lwjzj","openshift-network-diagnostics/network-check-target-bmdgm","kube-system/kube-apiserver-proxy-ip-10-0-136-192.ec2.internal"] Jul 07 17:54:13.270835 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.270810 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6zwkl" Jul 07 17:54:13.272551 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.272227 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:54:13.272551 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:13.272332 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmdgm" podUID="8a7f4631-8582-4636-832f-e026bb298b27" Jul 07 17:54:13.273876 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.273849 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 07 17:54:13.274265 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.274238 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 07 17:54:13.274394 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.274332 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wfrk" Jul 07 17:54:13.274394 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.274380 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vlwzk\"" Jul 07 17:54:13.275292 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.275269 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lbwbm" Jul 07 17:54:13.276881 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.276858 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 07 17:54:13.277015 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.276904 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.277807 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.277781 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 07 17:54:13.277900 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.277879 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 07 17:54:13.278275 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.278256 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 07 17:54:13.278345 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.278319 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 07 17:54:13.278345 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.278334 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8zjgn\"" Jul 07 17:54:13.278459 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.278357 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4spjk\"" Jul 07 17:54:13.278586 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.278565 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 07 17:54:13.278703 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.278651 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:13.278768 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:13.278738 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqq9h" podUID="3a1d71c1-6f70-441c-87bb-1f997df1fd5d" Jul 07 17:54:13.282954 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.282850 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 07 17:54:13.283296 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.283277 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 07 17:54:13.283971 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.283943 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 07 17:54:13.284073 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.284050 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dm6hm" Jul 07 17:54:13.284334 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.284313 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 07 17:54:13.285031 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.285010 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 07 17:54:13.285207 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.285190 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lww89\"" Jul 07 17:54:13.285346 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.285255 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 07 17:54:13.285659 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.285635 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 07 17:54:13.285659 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.285653 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.287186 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.287146 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p4mnt" Jul 07 17:54:13.288136 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.288113 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 07 17:54:13.289006 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.288776 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 07 17:54:13.289006 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.288839 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 07 17:54:13.289006 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.288849 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 07 17:54:13.289006 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.288861 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dg7d8\"" Jul 07 17:54:13.289261 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.289165 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9tgc8\"" Jul 07 17:54:13.289402 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.289367 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-49qq2" Jul 07 17:54:13.289504 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.289420 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 07 17:54:13.290127 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.290082 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 07 17:54:13.290791 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.290575 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 07 17:54:13.290791 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.290705 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mptkz\"" Jul 07 17:54:13.291131 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.291111 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.292825 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.292772 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:54:13.292923 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:13.292855 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwjzj" podUID="6e10d7dd-5837-417c-bbe3-2cbf4ca1c144" Jul 07 17:54:13.293961 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.293931 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 07 17:54:13.294048 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.294017 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 07 17:54:13.294226 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.294213 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kqgs8\"" Jul 07 17:54:13.294310 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.294294 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 07 17:54:13.294443 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.294429 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 07 17:54:13.294542 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.294526 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 07 17:54:13.294869 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.294850 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tkwrm\"" Jul 07 17:54:13.295038 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.295015 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 07 17:54:13.295326 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.295304 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-etc-sysctl-conf\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.295380 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.295345 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84hnh\" (UniqueName: \"kubernetes.io/projected/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-kube-api-access-84hnh\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.295425 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.295379 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a3e0ef7c-fa8d-47da-9d76-bfe681f296d6-iptables-alerter-script\") pod \"iptables-alerter-lbwbm\" (UID: \"a3e0ef7c-fa8d-47da-9d76-bfe681f296d6\") " pod="openshift-network-operator/iptables-alerter-lbwbm" Jul 07 17:54:13.295425 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.295406 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a1d71c1-6f70-441c-87bb-1f997df1fd5d-dbus\") pod \"global-pull-secret-syncer-xqq9h\" (UID: \"3a1d71c1-6f70-441c-87bb-1f997df1fd5d\") " pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:13.295498 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.295433 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-host-kubelet\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.295498 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.295462 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5c2d819c-e495-47ec-a1e7-4b2582a195bd-ovnkube-script-lib\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.295498 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.295484 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-run\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.295593 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.295529 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2lmc\" (UniqueName: \"kubernetes.io/projected/e5c3b9f0-ea6c-4262-b4e7-88945edc9f6f-kube-api-access-q2lmc\") pod \"node-resolver-p4mnt\" (UID: \"e5c3b9f0-ea6c-4262-b4e7-88945edc9f6f\") " pod="openshift-dns/node-resolver-p4mnt" Jul 07 17:54:13.295593 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.295566 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a1d71c1-6f70-441c-87bb-1f997df1fd5d-original-pull-secret\") pod \"global-pull-secret-syncer-xqq9h\" (UID: \"3a1d71c1-6f70-441c-87bb-1f997df1fd5d\") " pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:13.295656 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.295592 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-etc-modprobe-d\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.295656 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.295621 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-sys\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.295656 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.295648 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e5c3b9f0-ea6c-4262-b4e7-88945edc9f6f-hosts-file\") pod \"node-resolver-p4mnt\" (UID: \"e5c3b9f0-ea6c-4262-b4e7-88945edc9f6f\") " pod="openshift-dns/node-resolver-p4mnt" Jul 07 17:54:13.295748 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.295675 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1128d133-9305-4295-adec-94b196bd6f38-sys-fs\") pod \"aws-ebs-csi-driver-node-8wfrk\" (UID: \"1128d133-9305-4295-adec-94b196bd6f38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wfrk" Jul 07 17:54:13.295748 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.295702 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3e0ef7c-fa8d-47da-9d76-bfe681f296d6-host-slash\") pod \"iptables-alerter-lbwbm\" (UID: \"a3e0ef7c-fa8d-47da-9d76-bfe681f296d6\") " pod="openshift-network-operator/iptables-alerter-lbwbm" Jul 07 17:54:13.295748 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.295728 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1128d133-9305-4295-adec-94b196bd6f38-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8wfrk\" (UID: \"1128d133-9305-4295-adec-94b196bd6f38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wfrk" Jul 07 17:54:13.295840 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.295830 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f78b3750-ae77-4cf9-af3e-4ec59bc87dde-konnectivity-ca\") pod \"konnectivity-agent-6zwkl\" (UID: \"f78b3750-ae77-4cf9-af3e-4ec59bc87dde\") " pod="kube-system/konnectivity-agent-6zwkl" Jul 07 17:54:13.295873 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.295859 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-host-cni-bin\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.295925 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.295900 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a1d71c1-6f70-441c-87bb-1f997df1fd5d-kubelet-config\") pod \"global-pull-secret-syncer-xqq9h\" (UID: \"3a1d71c1-6f70-441c-87bb-1f997df1fd5d\") " pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:13.296067 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.295944 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-etc-sysconfig\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.296067 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.296002 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-host-run-netns\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.296067 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.296044 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-host\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.296237 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.296069 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5ea083a-f7fb-4e82-a02a-b930204910b6-host\") pod \"node-ca-dm6hm\" (UID: \"d5ea083a-f7fb-4e82-a02a-b930204910b6\") " pod="openshift-image-registry/node-ca-dm6hm" Jul 07 17:54:13.296237 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.296116 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-var-lib-openvswitch\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.296237 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.296142 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-lib-modules\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.296237 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.296187 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-tmp\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.296237 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.296213 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzv6d\" (UniqueName: \"kubernetes.io/projected/d5ea083a-f7fb-4e82-a02a-b930204910b6-kube-api-access-jzv6d\") pod \"node-ca-dm6hm\" (UID: \"d5ea083a-f7fb-4e82-a02a-b930204910b6\") " pod="openshift-image-registry/node-ca-dm6hm" Jul 07 17:54:13.296487 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.296256 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-systemd-units\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.296487 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.296280 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-host-slash\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.296601 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.296556 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-run-openvswitch\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.296657 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.296618 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-host-run-ovn-kubernetes\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.296711 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.296660 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5c2d819c-e495-47ec-a1e7-4b2582a195bd-env-overrides\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.296711 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.296689 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d5ea083a-f7fb-4e82-a02a-b930204910b6-serviceca\") pod \"node-ca-dm6hm\" (UID: \"d5ea083a-f7fb-4e82-a02a-b930204910b6\") " pod="openshift-image-registry/node-ca-dm6hm" Jul 07 17:54:13.296816 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.296734 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-run-systemd\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.296816 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.296787 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-log-socket\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.296918 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.296815 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1128d133-9305-4295-adec-94b196bd6f38-etc-selinux\") pod \"aws-ebs-csi-driver-node-8wfrk\" (UID: \"1128d133-9305-4295-adec-94b196bd6f38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wfrk" Jul 07 17:54:13.296918 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.296868 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c2d819c-e495-47ec-a1e7-4b2582a195bd-ovn-node-metrics-cert\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.297063 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.296902 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww42g\" (UniqueName: \"kubernetes.io/projected/1128d133-9305-4295-adec-94b196bd6f38-kube-api-access-ww42g\") pod \"aws-ebs-csi-driver-node-8wfrk\" (UID: \"1128d133-9305-4295-adec-94b196bd6f38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wfrk" Jul 07 17:54:13.297063 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.296955 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2fbb\" (UniqueName: \"kubernetes.io/projected/8a7f4631-8582-4636-832f-e026bb298b27-kube-api-access-g2fbb\") pod \"network-check-target-bmdgm\" (UID: \"8a7f4631-8582-4636-832f-e026bb298b27\") " pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:54:13.297063 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.296999 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-run-ovn\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.297205 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.297056 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-etc-sysctl-d\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.297205 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.297086 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5c3b9f0-ea6c-4262-b4e7-88945edc9f6f-tmp-dir\") pod \"node-resolver-p4mnt\" (UID: \"e5c3b9f0-ea6c-4262-b4e7-88945edc9f6f\") " pod="openshift-dns/node-resolver-p4mnt" Jul 07 17:54:13.297205 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.297121 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlw9x\" (UniqueName: \"kubernetes.io/projected/5c2d819c-e495-47ec-a1e7-4b2582a195bd-kube-api-access-hlw9x\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.297205 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.297176 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-etc-kubernetes\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.297330 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.297214 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-var-lib-kubelet\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.297330 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.297237 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-etc-tuned\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.297330 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.297262 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9pxt\" (UniqueName: \"kubernetes.io/projected/a3e0ef7c-fa8d-47da-9d76-bfe681f296d6-kube-api-access-w9pxt\") pod \"iptables-alerter-lbwbm\" (UID: \"a3e0ef7c-fa8d-47da-9d76-bfe681f296d6\") " pod="openshift-network-operator/iptables-alerter-lbwbm" Jul 07 17:54:13.297330 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.297298 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f78b3750-ae77-4cf9-af3e-4ec59bc87dde-agent-certs\") pod \"konnectivity-agent-6zwkl\" (UID: \"f78b3750-ae77-4cf9-af3e-4ec59bc87dde\") " pod="kube-system/konnectivity-agent-6zwkl" Jul 07 17:54:13.297330 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.297312 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-etc-openvswitch\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.297501 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.297327 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-node-log\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.297501 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.297351 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-etc-systemd\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.297501 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.297407 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1128d133-9305-4295-adec-94b196bd6f38-socket-dir\") pod \"aws-ebs-csi-driver-node-8wfrk\" (UID: \"1128d133-9305-4295-adec-94b196bd6f38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wfrk" Jul 07 17:54:13.297651 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.297511 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1128d133-9305-4295-adec-94b196bd6f38-registration-dir\") pod \"aws-ebs-csi-driver-node-8wfrk\" (UID: \"1128d133-9305-4295-adec-94b196bd6f38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wfrk" Jul 07 17:54:13.297651 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.297600 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-host-cni-netd\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.297762 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.297657 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.297762 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.297711 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5c2d819c-e495-47ec-a1e7-4b2582a195bd-ovnkube-config\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.297869 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.297768 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1128d133-9305-4295-adec-94b196bd6f38-device-dir\") pod \"aws-ebs-csi-driver-node-8wfrk\" (UID: \"1128d133-9305-4295-adec-94b196bd6f38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wfrk" Jul 07 17:54:13.299547 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.299523 2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 17:54:13.322096 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.322052 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cnl9j" Jul 07 17:54:13.334484 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.334075 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cnl9j" Jul 07 17:54:13.384683 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.384646 2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 17:54:13.385081 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.385061 2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 07 17:54:13.399013 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.398976 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1128d133-9305-4295-adec-94b196bd6f38-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8wfrk\" (UID: \"1128d133-9305-4295-adec-94b196bd6f38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wfrk" Jul 07 17:54:13.399179 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399029 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f78b3750-ae77-4cf9-af3e-4ec59bc87dde-konnectivity-ca\") pod \"konnectivity-agent-6zwkl\" (UID: \"f78b3750-ae77-4cf9-af3e-4ec59bc87dde\") " pod="kube-system/konnectivity-agent-6zwkl" Jul 07 17:54:13.399179 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399061 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-host-cni-bin\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.399179 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399088 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a1d71c1-6f70-441c-87bb-1f997df1fd5d-kubelet-config\") pod \"global-pull-secret-syncer-xqq9h\" (UID: \"3a1d71c1-6f70-441c-87bb-1f997df1fd5d\") " pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:13.399179 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399120 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/174cb3d6-9171-48b6-8e00-d094fa4b4757-multus-daemon-config\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.399179 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399149 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/797a1f8a-8d68-4c01-a38b-67d405653bd0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-49qq2\" (UID: \"797a1f8a-8d68-4c01-a38b-67d405653bd0\") " pod="openshift-multus/multus-additional-cni-plugins-49qq2" Jul 07 17:54:13.399179 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399177 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-etc-sysconfig\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.399511 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399203 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-host-var-lib-kubelet\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.399511 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399229 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-host-run-netns\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.399511 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399259 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-host\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.399511 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399284 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5ea083a-f7fb-4e82-a02a-b930204910b6-host\") pod \"node-ca-dm6hm\" (UID: \"d5ea083a-f7fb-4e82-a02a-b930204910b6\") " pod="openshift-image-registry/node-ca-dm6hm" Jul 07 17:54:13.399511 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399312 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-system-cni-dir\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.399511 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399336 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-host-var-lib-cni-multus\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.399511 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399362 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/797a1f8a-8d68-4c01-a38b-67d405653bd0-system-cni-dir\") pod \"multus-additional-cni-plugins-49qq2\" (UID: \"797a1f8a-8d68-4c01-a38b-67d405653bd0\") " pod="openshift-multus/multus-additional-cni-plugins-49qq2" Jul 07 17:54:13.399511 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399389 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-var-lib-openvswitch\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.399511 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399414 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-lib-modules\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.399511 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399439 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-tmp\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.399511 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399463 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzv6d\" (UniqueName: \"kubernetes.io/projected/d5ea083a-f7fb-4e82-a02a-b930204910b6-kube-api-access-jzv6d\") pod \"node-ca-dm6hm\" (UID: \"d5ea083a-f7fb-4e82-a02a-b930204910b6\") " pod="openshift-image-registry/node-ca-dm6hm" Jul 07 17:54:13.399511 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399486 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-systemd-units\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.399511 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399501 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-host-slash\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.399511 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399517 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-run-openvswitch\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.400233 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399535 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-host-run-ovn-kubernetes\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.400233 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399562 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5c2d819c-e495-47ec-a1e7-4b2582a195bd-env-overrides\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.400233 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399586 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d5ea083a-f7fb-4e82-a02a-b930204910b6-serviceca\") pod \"node-ca-dm6hm\" (UID: \"d5ea083a-f7fb-4e82-a02a-b930204910b6\") " pod="openshift-image-registry/node-ca-dm6hm" Jul 07 17:54:13.400233 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399611 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-run-systemd\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.400233 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399636 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-log-socket\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.400233 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399668 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1128d133-9305-4295-adec-94b196bd6f38-etc-selinux\") pod \"aws-ebs-csi-driver-node-8wfrk\" (UID: \"1128d133-9305-4295-adec-94b196bd6f38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wfrk" Jul 07 17:54:13.400233 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399692 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-multus-cni-dir\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.400233 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399716 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-os-release\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.400233 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399742 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-host-run-k8s-cni-cncf-io\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.400233 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399769 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/797a1f8a-8d68-4c01-a38b-67d405653bd0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-49qq2\" (UID: \"797a1f8a-8d68-4c01-a38b-67d405653bd0\") " pod="openshift-multus/multus-additional-cni-plugins-49qq2" Jul 07 17:54:13.400233 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399797 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c2d819c-e495-47ec-a1e7-4b2582a195bd-ovn-node-metrics-cert\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.400233 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399829 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww42g\" (UniqueName: \"kubernetes.io/projected/1128d133-9305-4295-adec-94b196bd6f38-kube-api-access-ww42g\") pod \"aws-ebs-csi-driver-node-8wfrk\" (UID: \"1128d133-9305-4295-adec-94b196bd6f38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wfrk" Jul 07 17:54:13.400233 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399855 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-host-run-multus-certs\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.400233 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399894 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2fbb\" (UniqueName: \"kubernetes.io/projected/8a7f4631-8582-4636-832f-e026bb298b27-kube-api-access-g2fbb\") pod \"network-check-target-bmdgm\" (UID: \"8a7f4631-8582-4636-832f-e026bb298b27\") " pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:54:13.400233 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399921 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-run-ovn\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.400233 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.399947 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-etc-sysctl-d\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.400233 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400001 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5c3b9f0-ea6c-4262-b4e7-88945edc9f6f-tmp-dir\") pod \"node-resolver-p4mnt\" (UID: \"e5c3b9f0-ea6c-4262-b4e7-88945edc9f6f\") " pod="openshift-dns/node-resolver-p4mnt" Jul 07 17:54:13.400832 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400044 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-cnibin\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.400832 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400070 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-multus-socket-dir-parent\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.400832 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400096 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-hostroot\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.400832 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400122 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-multus-conf-dir\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.400832 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400149 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlw9x\" (UniqueName: \"kubernetes.io/projected/5c2d819c-e495-47ec-a1e7-4b2582a195bd-kube-api-access-hlw9x\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.400832 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400178 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-etc-kubernetes\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.400832 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400208 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-var-lib-kubelet\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.400832 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400242 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-etc-tuned\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.400832 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400272 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9pxt\" (UniqueName: \"kubernetes.io/projected/a3e0ef7c-fa8d-47da-9d76-bfe681f296d6-kube-api-access-w9pxt\") pod \"iptables-alerter-lbwbm\" (UID: \"a3e0ef7c-fa8d-47da-9d76-bfe681f296d6\") " pod="openshift-network-operator/iptables-alerter-lbwbm" Jul 07 17:54:13.400832 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400299 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/797a1f8a-8d68-4c01-a38b-67d405653bd0-os-release\") pod \"multus-additional-cni-plugins-49qq2\" (UID: \"797a1f8a-8d68-4c01-a38b-67d405653bd0\") " pod="openshift-multus/multus-additional-cni-plugins-49qq2" Jul 07 17:54:13.400832 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400325 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/797a1f8a-8d68-4c01-a38b-67d405653bd0-cni-binary-copy\") pod \"multus-additional-cni-plugins-49qq2\" (UID: \"797a1f8a-8d68-4c01-a38b-67d405653bd0\") " pod="openshift-multus/multus-additional-cni-plugins-49qq2" Jul 07 17:54:13.400832 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400385 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f78b3750-ae77-4cf9-af3e-4ec59bc87dde-agent-certs\") pod \"konnectivity-agent-6zwkl\" (UID: \"f78b3750-ae77-4cf9-af3e-4ec59bc87dde\") " pod="kube-system/konnectivity-agent-6zwkl" Jul 07 17:54:13.400832 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400412 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-etc-openvswitch\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.400832 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400429 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-node-log\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.400832 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400450 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-etc-systemd\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.400832 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400466 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1128d133-9305-4295-adec-94b196bd6f38-socket-dir\") pod \"aws-ebs-csi-driver-node-8wfrk\" (UID: \"1128d133-9305-4295-adec-94b196bd6f38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wfrk" Jul 07 17:54:13.400832 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400487 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1128d133-9305-4295-adec-94b196bd6f38-registration-dir\") pod \"aws-ebs-csi-driver-node-8wfrk\" (UID: \"1128d133-9305-4295-adec-94b196bd6f38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wfrk" Jul 07 17:54:13.401586 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400515 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-host-run-netns\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.401586 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400543 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7d79\" (UniqueName: \"kubernetes.io/projected/174cb3d6-9171-48b6-8e00-d094fa4b4757-kube-api-access-k7d79\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.401586 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400569 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-host-cni-netd\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.401586 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400598 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.401586 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400625 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5c2d819c-e495-47ec-a1e7-4b2582a195bd-ovnkube-config\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.401586 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400657 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1128d133-9305-4295-adec-94b196bd6f38-device-dir\") pod \"aws-ebs-csi-driver-node-8wfrk\" (UID: \"1128d133-9305-4295-adec-94b196bd6f38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wfrk" Jul 07 17:54:13.401586 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400684 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hdb9\" (UniqueName: \"kubernetes.io/projected/6e10d7dd-5837-417c-bbe3-2cbf4ca1c144-kube-api-access-6hdb9\") pod \"network-metrics-daemon-lwjzj\" (UID: \"6e10d7dd-5837-417c-bbe3-2cbf4ca1c144\") " pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:54:13.401586 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400709 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/797a1f8a-8d68-4c01-a38b-67d405653bd0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-49qq2\" (UID: \"797a1f8a-8d68-4c01-a38b-67d405653bd0\") " pod="openshift-multus/multus-additional-cni-plugins-49qq2" Jul 07 17:54:13.401586 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400727 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-etc-sysctl-conf\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.401586 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400748 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84hnh\" (UniqueName: \"kubernetes.io/projected/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-kube-api-access-84hnh\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.401586 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400806 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a3e0ef7c-fa8d-47da-9d76-bfe681f296d6-iptables-alerter-script\") pod \"iptables-alerter-lbwbm\" (UID: \"a3e0ef7c-fa8d-47da-9d76-bfe681f296d6\") " pod="openshift-network-operator/iptables-alerter-lbwbm" Jul 07 17:54:13.401586 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400835 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a1d71c1-6f70-441c-87bb-1f997df1fd5d-dbus\") pod \"global-pull-secret-syncer-xqq9h\" (UID: \"3a1d71c1-6f70-441c-87bb-1f997df1fd5d\") " pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:13.401586 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400862 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/174cb3d6-9171-48b6-8e00-d094fa4b4757-cni-binary-copy\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.401586 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400909 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e10d7dd-5837-417c-bbe3-2cbf4ca1c144-metrics-certs\") pod \"network-metrics-daemon-lwjzj\" (UID: \"6e10d7dd-5837-417c-bbe3-2cbf4ca1c144\") " pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:54:13.401586 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400942 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-host-kubelet\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.401586 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.400993 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5c2d819c-e495-47ec-a1e7-4b2582a195bd-ovnkube-script-lib\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.401586 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.401018 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-run\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.402447 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.401045 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2lmc\" (UniqueName: \"kubernetes.io/projected/e5c3b9f0-ea6c-4262-b4e7-88945edc9f6f-kube-api-access-q2lmc\") pod \"node-resolver-p4mnt\" (UID: \"e5c3b9f0-ea6c-4262-b4e7-88945edc9f6f\") " pod="openshift-dns/node-resolver-p4mnt" Jul 07 17:54:13.402447 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.401112 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a1d71c1-6f70-441c-87bb-1f997df1fd5d-original-pull-secret\") pod \"global-pull-secret-syncer-xqq9h\" (UID: \"3a1d71c1-6f70-441c-87bb-1f997df1fd5d\") " pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:13.402447 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.401115 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1128d133-9305-4295-adec-94b196bd6f38-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8wfrk\" (UID: \"1128d133-9305-4295-adec-94b196bd6f38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wfrk" Jul 07 17:54:13.402447 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.401144 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-etc-kubernetes\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.402447 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.401170 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-etc-modprobe-d\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.402447 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.401198 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-sys\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.402447 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.401223 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-host-var-lib-cni-bin\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.402447 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.401268 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/797a1f8a-8d68-4c01-a38b-67d405653bd0-cnibin\") pod \"multus-additional-cni-plugins-49qq2\" (UID: \"797a1f8a-8d68-4c01-a38b-67d405653bd0\") " pod="openshift-multus/multus-additional-cni-plugins-49qq2" Jul 07 17:54:13.402447 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.401295 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47x7t\" (UniqueName: \"kubernetes.io/projected/797a1f8a-8d68-4c01-a38b-67d405653bd0-kube-api-access-47x7t\") pod \"multus-additional-cni-plugins-49qq2\" (UID: \"797a1f8a-8d68-4c01-a38b-67d405653bd0\") " pod="openshift-multus/multus-additional-cni-plugins-49qq2" Jul 07 17:54:13.402447 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.401328 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e5c3b9f0-ea6c-4262-b4e7-88945edc9f6f-hosts-file\") pod \"node-resolver-p4mnt\" (UID: \"e5c3b9f0-ea6c-4262-b4e7-88945edc9f6f\") " pod="openshift-dns/node-resolver-p4mnt" Jul 07 17:54:13.402447 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.401443 2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 07 17:54:13.402447 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.401461 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1128d133-9305-4295-adec-94b196bd6f38-sys-fs\") pod \"aws-ebs-csi-driver-node-8wfrk\" (UID: \"1128d133-9305-4295-adec-94b196bd6f38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wfrk" Jul 07 17:54:13.402447 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.401465 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-systemd-units\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.402447 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.401520 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-host-slash\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.402447 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.401570 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-run-ovn\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.402447 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.401577 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-run-openvswitch\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.402447 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.401615 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3e0ef7c-fa8d-47da-9d76-bfe681f296d6-host-slash\") pod \"iptables-alerter-lbwbm\" (UID: \"a3e0ef7c-fa8d-47da-9d76-bfe681f296d6\") " pod="openshift-network-operator/iptables-alerter-lbwbm" Jul 07 17:54:13.402447 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.401754 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-etc-sysctl-d\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.403497 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.401804 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-host-run-ovn-kubernetes\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.403497 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.401820 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f78b3750-ae77-4cf9-af3e-4ec59bc87dde-konnectivity-ca\") pod \"konnectivity-agent-6zwkl\" (UID: \"f78b3750-ae77-4cf9-af3e-4ec59bc87dde\") " pod="kube-system/konnectivity-agent-6zwkl" Jul 07 17:54:13.403497 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.401894 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5c3b9f0-ea6c-4262-b4e7-88945edc9f6f-tmp-dir\") pod \"node-resolver-p4mnt\" (UID: \"e5c3b9f0-ea6c-4262-b4e7-88945edc9f6f\") " pod="openshift-dns/node-resolver-p4mnt" Jul 07 17:54:13.403497 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.401909 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-host-cni-bin\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.403497 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.401957 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a1d71c1-6f70-441c-87bb-1f997df1fd5d-kubelet-config\") pod \"global-pull-secret-syncer-xqq9h\" (UID: \"3a1d71c1-6f70-441c-87bb-1f997df1fd5d\") " pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:13.403497 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.402030 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-etc-sysconfig\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.403497 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.402069 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-host-run-netns\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.403497 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.402109 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-host\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.403497 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.402138 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5ea083a-f7fb-4e82-a02a-b930204910b6-host\") pod \"node-ca-dm6hm\" (UID: \"d5ea083a-f7fb-4e82-a02a-b930204910b6\") " pod="openshift-image-registry/node-ca-dm6hm" Jul 07 17:54:13.403497 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.402187 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-etc-kubernetes\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.403497 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.402283 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-etc-sysctl-conf\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.403497 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.402311 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5c2d819c-e495-47ec-a1e7-4b2582a195bd-env-overrides\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.403497 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.402322 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-var-lib-kubelet\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.403497 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:13.402338 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:54:13.403497 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:13.402441 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a1d71c1-6f70-441c-87bb-1f997df1fd5d-original-pull-secret podName:3a1d71c1-6f70-441c-87bb-1f997df1fd5d nodeName:}" failed. No retries permitted until 2026-07-07 17:54:13.902399125 +0000 UTC m=+2.985002822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a1d71c1-6f70-441c-87bb-1f997df1fd5d-original-pull-secret") pod "global-pull-secret-syncer-xqq9h" (UID: "3a1d71c1-6f70-441c-87bb-1f997df1fd5d") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:54:13.403497 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.402478 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-etc-modprobe-d\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.403497 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.402541 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-sys\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.403497 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.402773 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-var-lib-openvswitch\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.404386 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.402799 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d5ea083a-f7fb-4e82-a02a-b930204910b6-serviceca\") pod \"node-ca-dm6hm\" (UID: \"d5ea083a-f7fb-4e82-a02a-b930204910b6\") " pod="openshift-image-registry/node-ca-dm6hm" Jul 07 17:54:13.404386 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.402783 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1128d133-9305-4295-adec-94b196bd6f38-sys-fs\") pod \"aws-ebs-csi-driver-node-8wfrk\" (UID: \"1128d133-9305-4295-adec-94b196bd6f38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wfrk" Jul 07 17:54:13.404386 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.402817 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e5c3b9f0-ea6c-4262-b4e7-88945edc9f6f-hosts-file\") pod \"node-resolver-p4mnt\" (UID: \"e5c3b9f0-ea6c-4262-b4e7-88945edc9f6f\") " pod="openshift-dns/node-resolver-p4mnt" Jul 07 17:54:13.404386 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.402832 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-host-kubelet\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.404386 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.402925 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-run\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.404386 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.403044 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-lib-modules\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.404386 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.401521 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3e0ef7c-fa8d-47da-9d76-bfe681f296d6-host-slash\") pod \"iptables-alerter-lbwbm\" (UID: \"a3e0ef7c-fa8d-47da-9d76-bfe681f296d6\") " pod="openshift-network-operator/iptables-alerter-lbwbm" Jul 07 17:54:13.404386 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.403245 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a3e0ef7c-fa8d-47da-9d76-bfe681f296d6-iptables-alerter-script\") pod \"iptables-alerter-lbwbm\" (UID: \"a3e0ef7c-fa8d-47da-9d76-bfe681f296d6\") " pod="openshift-network-operator/iptables-alerter-lbwbm" Jul 07 17:54:13.404386 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.403298 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1128d133-9305-4295-adec-94b196bd6f38-socket-dir\") pod \"aws-ebs-csi-driver-node-8wfrk\" (UID: \"1128d133-9305-4295-adec-94b196bd6f38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wfrk" Jul 07 17:54:13.404386 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.403368 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-log-socket\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.404386 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.403394 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-node-log\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.404386 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.403435 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5c2d819c-e495-47ec-a1e7-4b2582a195bd-ovnkube-script-lib\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.404386 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.403478 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-etc-systemd\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.404386 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.403540 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a1d71c1-6f70-441c-87bb-1f997df1fd5d-dbus\") pod \"global-pull-secret-syncer-xqq9h\" (UID: \"3a1d71c1-6f70-441c-87bb-1f997df1fd5d\") " pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:13.404386 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.403546 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1128d133-9305-4295-adec-94b196bd6f38-registration-dir\") pod \"aws-ebs-csi-driver-node-8wfrk\" (UID: \"1128d133-9305-4295-adec-94b196bd6f38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wfrk" Jul 07 17:54:13.404386 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.403569 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-etc-openvswitch\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.404386 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.403592 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-host-cni-netd\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.404386 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.403675 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.404962 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.403681 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1128d133-9305-4295-adec-94b196bd6f38-device-dir\") pod \"aws-ebs-csi-driver-node-8wfrk\" (UID: \"1128d133-9305-4295-adec-94b196bd6f38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wfrk" Jul 07 17:54:13.404962 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.403679 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1128d133-9305-4295-adec-94b196bd6f38-etc-selinux\") pod \"aws-ebs-csi-driver-node-8wfrk\" (UID: \"1128d133-9305-4295-adec-94b196bd6f38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wfrk" Jul 07 17:54:13.404962 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.403727 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5c2d819c-e495-47ec-a1e7-4b2582a195bd-run-systemd\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.404962 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.404230 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5c2d819c-e495-47ec-a1e7-4b2582a195bd-ovnkube-config\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.406327 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.406302 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-tmp\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.406465 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.406361 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-etc-tuned\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.407546 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.407520 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f78b3750-ae77-4cf9-af3e-4ec59bc87dde-agent-certs\") pod \"konnectivity-agent-6zwkl\" (UID: \"f78b3750-ae77-4cf9-af3e-4ec59bc87dde\") " pod="kube-system/konnectivity-agent-6zwkl" Jul 07 17:54:13.407705 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:13.407607 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:54:13.407705 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:13.407634 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:54:13.407705 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:13.407652 2583 projected.go:196] Error preparing data for projected volume kube-api-access-g2fbb for pod openshift-network-diagnostics/network-check-target-bmdgm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:54:13.407935 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:13.407726 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a7f4631-8582-4636-832f-e026bb298b27-kube-api-access-g2fbb podName:8a7f4631-8582-4636-832f-e026bb298b27 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:13.907704007 +0000 UTC m=+2.990307691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-g2fbb" (UniqueName: "kubernetes.io/projected/8a7f4631-8582-4636-832f-e026bb298b27-kube-api-access-g2fbb") pod "network-check-target-bmdgm" (UID: "8a7f4631-8582-4636-832f-e026bb298b27") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:54:13.407935 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.407917 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c2d819c-e495-47ec-a1e7-4b2582a195bd-ovn-node-metrics-cert\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.411475 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.411442 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzv6d\" (UniqueName: \"kubernetes.io/projected/d5ea083a-f7fb-4e82-a02a-b930204910b6-kube-api-access-jzv6d\") pod \"node-ca-dm6hm\" (UID: \"d5ea083a-f7fb-4e82-a02a-b930204910b6\") " pod="openshift-image-registry/node-ca-dm6hm" Jul 07 17:54:13.412886 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.412859 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlw9x\" (UniqueName: \"kubernetes.io/projected/5c2d819c-e495-47ec-a1e7-4b2582a195bd-kube-api-access-hlw9x\") pod \"ovnkube-node-zjzpw\" (UID: \"5c2d819c-e495-47ec-a1e7-4b2582a195bd\") " pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.413220 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.413194 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84hnh\" (UniqueName: \"kubernetes.io/projected/f09531d6-f2e3-47bb-acb2-de1092f4f9ff-kube-api-access-84hnh\") pod \"tuned-t2g29\" (UID: \"f09531d6-f2e3-47bb-acb2-de1092f4f9ff\") " pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.414965 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.414910 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww42g\" (UniqueName: \"kubernetes.io/projected/1128d133-9305-4295-adec-94b196bd6f38-kube-api-access-ww42g\") pod \"aws-ebs-csi-driver-node-8wfrk\" (UID: \"1128d133-9305-4295-adec-94b196bd6f38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wfrk" Jul 07 17:54:13.415585 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.415561 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9pxt\" (UniqueName: \"kubernetes.io/projected/a3e0ef7c-fa8d-47da-9d76-bfe681f296d6-kube-api-access-w9pxt\") pod \"iptables-alerter-lbwbm\" (UID: \"a3e0ef7c-fa8d-47da-9d76-bfe681f296d6\") " pod="openshift-network-operator/iptables-alerter-lbwbm" Jul 07 17:54:13.417430 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.417403 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2lmc\" (UniqueName: \"kubernetes.io/projected/e5c3b9f0-ea6c-4262-b4e7-88945edc9f6f-kube-api-access-q2lmc\") pod \"node-resolver-p4mnt\" (UID: \"e5c3b9f0-ea6c-4262-b4e7-88945edc9f6f\") " pod="openshift-dns/node-resolver-p4mnt" Jul 07 17:54:13.504129 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.504081 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-cnibin\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.504129 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.504133 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-multus-socket-dir-parent\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.504395 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.504151 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-hostroot\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.504395 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.504166 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-multus-conf-dir\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.504395 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.504187 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/797a1f8a-8d68-4c01-a38b-67d405653bd0-os-release\") pod \"multus-additional-cni-plugins-49qq2\" (UID: \"797a1f8a-8d68-4c01-a38b-67d405653bd0\") " pod="openshift-multus/multus-additional-cni-plugins-49qq2" Jul 07 17:54:13.504395 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.504209 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/797a1f8a-8d68-4c01-a38b-67d405653bd0-cni-binary-copy\") pod \"multus-additional-cni-plugins-49qq2\" (UID: \"797a1f8a-8d68-4c01-a38b-67d405653bd0\") " pod="openshift-multus/multus-additional-cni-plugins-49qq2" Jul 07 17:54:13.504395 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.504234 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-host-run-netns\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.504395 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.504236 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-hostroot\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.504395 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.504239 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-multus-socket-dir-parent\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.504395 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.504265 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7d79\" (UniqueName: \"kubernetes.io/projected/174cb3d6-9171-48b6-8e00-d094fa4b4757-kube-api-access-k7d79\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.504395 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.504244 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-multus-conf-dir\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.504395 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.504287 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-cnibin\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.504395 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.504289 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-host-run-netns\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.504395 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.504318 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hdb9\" (UniqueName: \"kubernetes.io/projected/6e10d7dd-5837-417c-bbe3-2cbf4ca1c144-kube-api-access-6hdb9\") pod \"network-metrics-daemon-lwjzj\" (UID: \"6e10d7dd-5837-417c-bbe3-2cbf4ca1c144\") " pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:54:13.504395 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.504307 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/797a1f8a-8d68-4c01-a38b-67d405653bd0-os-release\") pod \"multus-additional-cni-plugins-49qq2\" (UID: \"797a1f8a-8d68-4c01-a38b-67d405653bd0\") " pod="openshift-multus/multus-additional-cni-plugins-49qq2" Jul 07 17:54:13.504922 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.504408 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/797a1f8a-8d68-4c01-a38b-67d405653bd0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-49qq2\" (UID: \"797a1f8a-8d68-4c01-a38b-67d405653bd0\") " pod="openshift-multus/multus-additional-cni-plugins-49qq2" Jul 07 17:54:13.504922 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.504535 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/174cb3d6-9171-48b6-8e00-d094fa4b4757-cni-binary-copy\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.504922 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.504567 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e10d7dd-5837-417c-bbe3-2cbf4ca1c144-metrics-certs\") pod \"network-metrics-daemon-lwjzj\" (UID: \"6e10d7dd-5837-417c-bbe3-2cbf4ca1c144\") " pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:54:13.504922 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.504610 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-etc-kubernetes\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.504922 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.504639 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-host-var-lib-cni-bin\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.504922 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:13.504719 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:54:13.504922 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.504766 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-etc-kubernetes\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.504922 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.504794 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/797a1f8a-8d68-4c01-a38b-67d405653bd0-cnibin\") pod \"multus-additional-cni-plugins-49qq2\" (UID: \"797a1f8a-8d68-4c01-a38b-67d405653bd0\") " pod="openshift-multus/multus-additional-cni-plugins-49qq2" Jul 07 17:54:13.504922 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.504822 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47x7t\" (UniqueName: \"kubernetes.io/projected/797a1f8a-8d68-4c01-a38b-67d405653bd0-kube-api-access-47x7t\") pod \"multus-additional-cni-plugins-49qq2\" (UID: \"797a1f8a-8d68-4c01-a38b-67d405653bd0\") " pod="openshift-multus/multus-additional-cni-plugins-49qq2" Jul 07 17:54:13.504922 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.504904 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/797a1f8a-8d68-4c01-a38b-67d405653bd0-cni-binary-copy\") pod \"multus-additional-cni-plugins-49qq2\" (UID: \"797a1f8a-8d68-4c01-a38b-67d405653bd0\") " pod="openshift-multus/multus-additional-cni-plugins-49qq2" Jul 07 17:54:13.505317 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.504915 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/174cb3d6-9171-48b6-8e00-d094fa4b4757-multus-daemon-config\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.505317 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.505027 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/797a1f8a-8d68-4c01-a38b-67d405653bd0-cnibin\") pod \"multus-additional-cni-plugins-49qq2\" (UID: \"797a1f8a-8d68-4c01-a38b-67d405653bd0\") " pod="openshift-multus/multus-additional-cni-plugins-49qq2" Jul 07 17:54:13.505317 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.505041 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/797a1f8a-8d68-4c01-a38b-67d405653bd0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-49qq2\" (UID: \"797a1f8a-8d68-4c01-a38b-67d405653bd0\") " pod="openshift-multus/multus-additional-cni-plugins-49qq2" Jul 07 17:54:13.505317 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.505075 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-host-var-lib-cni-bin\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.505317 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.505104 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-host-var-lib-kubelet\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.505317 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.505113 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/797a1f8a-8d68-4c01-a38b-67d405653bd0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-49qq2\" (UID: \"797a1f8a-8d68-4c01-a38b-67d405653bd0\") " pod="openshift-multus/multus-additional-cni-plugins-49qq2" Jul 07 17:54:13.505317 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:13.505138 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e10d7dd-5837-417c-bbe3-2cbf4ca1c144-metrics-certs podName:6e10d7dd-5837-417c-bbe3-2cbf4ca1c144 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:14.005099045 +0000 UTC m=+3.087702735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e10d7dd-5837-417c-bbe3-2cbf4ca1c144-metrics-certs") pod "network-metrics-daemon-lwjzj" (UID: "6e10d7dd-5837-417c-bbe3-2cbf4ca1c144") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:54:13.505317 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.505144 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-host-var-lib-kubelet\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.505317 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.505164 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/174cb3d6-9171-48b6-8e00-d094fa4b4757-cni-binary-copy\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.505317 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.505178 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-system-cni-dir\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.505317 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.505207 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-host-var-lib-cni-multus\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.505317 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.505238 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/797a1f8a-8d68-4c01-a38b-67d405653bd0-system-cni-dir\") pod \"multus-additional-cni-plugins-49qq2\" (UID: \"797a1f8a-8d68-4c01-a38b-67d405653bd0\") " pod="openshift-multus/multus-additional-cni-plugins-49qq2" Jul 07 17:54:13.505317 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.505284 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-system-cni-dir\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.505317 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.505287 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-multus-cni-dir\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.505317 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.505321 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-host-var-lib-cni-multus\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.506018 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.505350 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-os-release\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.506018 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.505385 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/797a1f8a-8d68-4c01-a38b-67d405653bd0-system-cni-dir\") pod \"multus-additional-cni-plugins-49qq2\" (UID: \"797a1f8a-8d68-4c01-a38b-67d405653bd0\") " pod="openshift-multus/multus-additional-cni-plugins-49qq2" Jul 07 17:54:13.506018 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.505398 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-host-run-k8s-cni-cncf-io\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.506018 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.505426 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/797a1f8a-8d68-4c01-a38b-67d405653bd0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-49qq2\" (UID: \"797a1f8a-8d68-4c01-a38b-67d405653bd0\") " pod="openshift-multus/multus-additional-cni-plugins-49qq2" Jul 07 17:54:13.506018 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.505343 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-multus-cni-dir\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.506018 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.505438 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-host-run-k8s-cni-cncf-io\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.506018 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.505428 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-os-release\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.506018 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.505479 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-host-run-multus-certs\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.506018 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.505522 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/174cb3d6-9171-48b6-8e00-d094fa4b4757-multus-daemon-config\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.506018 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.505585 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/797a1f8a-8d68-4c01-a38b-67d405653bd0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-49qq2\" (UID: \"797a1f8a-8d68-4c01-a38b-67d405653bd0\") " pod="openshift-multus/multus-additional-cni-plugins-49qq2" Jul 07 17:54:13.506018 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.505601 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/174cb3d6-9171-48b6-8e00-d094fa4b4757-host-run-multus-certs\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.506018 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.505636 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/797a1f8a-8d68-4c01-a38b-67d405653bd0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-49qq2\" (UID: \"797a1f8a-8d68-4c01-a38b-67d405653bd0\") " pod="openshift-multus/multus-additional-cni-plugins-49qq2" Jul 07 17:54:13.520468 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.520352 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hdb9\" (UniqueName: \"kubernetes.io/projected/6e10d7dd-5837-417c-bbe3-2cbf4ca1c144-kube-api-access-6hdb9\") pod \"network-metrics-daemon-lwjzj\" (UID: \"6e10d7dd-5837-417c-bbe3-2cbf4ca1c144\") " pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:54:13.520624 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.520476 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7d79\" (UniqueName: \"kubernetes.io/projected/174cb3d6-9171-48b6-8e00-d094fa4b4757-kube-api-access-k7d79\") pod \"multus-z68qr\" (UID: \"174cb3d6-9171-48b6-8e00-d094fa4b4757\") " pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.520901 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.520872 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47x7t\" (UniqueName: \"kubernetes.io/projected/797a1f8a-8d68-4c01-a38b-67d405653bd0-kube-api-access-47x7t\") pod \"multus-additional-cni-plugins-49qq2\" (UID: \"797a1f8a-8d68-4c01-a38b-67d405653bd0\") " pod="openshift-multus/multus-additional-cni-plugins-49qq2" Jul 07 17:54:13.589311 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.589270 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6zwkl" Jul 07 17:54:13.605875 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.605822 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wfrk" Jul 07 17:54:13.617096 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.617054 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lbwbm" Jul 07 17:54:13.625408 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.625366 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:13.635567 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.635530 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dm6hm" Jul 07 17:54:13.645056 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.645020 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t2g29" Jul 07 17:54:13.654268 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.654240 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p4mnt" Jul 07 17:54:13.662260 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.662229 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-49qq2" Jul 07 17:54:13.670430 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.670389 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z68qr" Jul 07 17:54:13.908537 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.908494 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a1d71c1-6f70-441c-87bb-1f997df1fd5d-original-pull-secret\") pod \"global-pull-secret-syncer-xqq9h\" (UID: \"3a1d71c1-6f70-441c-87bb-1f997df1fd5d\") " pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:13.908754 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:13.908579 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2fbb\" (UniqueName: \"kubernetes.io/projected/8a7f4631-8582-4636-832f-e026bb298b27-kube-api-access-g2fbb\") pod \"network-check-target-bmdgm\" (UID: \"8a7f4631-8582-4636-832f-e026bb298b27\") " pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:54:13.908754 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:13.908686 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:54:13.908754 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:13.908731 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:54:13.908754 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:13.908753 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:54:13.908977 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:13.908771 2583 projected.go:196] Error preparing data for projected volume kube-api-access-g2fbb for pod openshift-network-diagnostics/network-check-target-bmdgm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:54:13.908977 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:13.908775 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a1d71c1-6f70-441c-87bb-1f997df1fd5d-original-pull-secret podName:3a1d71c1-6f70-441c-87bb-1f997df1fd5d nodeName:}" failed. No retries permitted until 2026-07-07 17:54:14.908751729 +0000 UTC m=+3.991355418 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a1d71c1-6f70-441c-87bb-1f997df1fd5d-original-pull-secret") pod "global-pull-secret-syncer-xqq9h" (UID: "3a1d71c1-6f70-441c-87bb-1f997df1fd5d") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:54:13.908977 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:13.908828 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a7f4631-8582-4636-832f-e026bb298b27-kube-api-access-g2fbb podName:8a7f4631-8582-4636-832f-e026bb298b27 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:14.908808793 +0000 UTC m=+3.991412476 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-g2fbb" (UniqueName: "kubernetes.io/projected/8a7f4631-8582-4636-832f-e026bb298b27-kube-api-access-g2fbb") pod "network-check-target-bmdgm" (UID: "8a7f4631-8582-4636-832f-e026bb298b27") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:54:14.008960 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:14.008900 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e10d7dd-5837-417c-bbe3-2cbf4ca1c144-metrics-certs\") pod \"network-metrics-daemon-lwjzj\" (UID: \"6e10d7dd-5837-417c-bbe3-2cbf4ca1c144\") " pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:54:14.009157 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:14.009073 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:54:14.009202 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:14.009174 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e10d7dd-5837-417c-bbe3-2cbf4ca1c144-metrics-certs podName:6e10d7dd-5837-417c-bbe3-2cbf4ca1c144 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:15.009149848 +0000 UTC m=+4.091753550 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e10d7dd-5837-417c-bbe3-2cbf4ca1c144-metrics-certs") pod "network-metrics-daemon-lwjzj" (UID: "6e10d7dd-5837-417c-bbe3-2cbf4ca1c144") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:54:14.337282 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:14.337173 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 17:49:13 +0000 UTC" deadline="2028-02-21 21:52:34.592569755 +0000 UTC" Jul 07 17:54:14.337282 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:14.337228 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14259h58m20.255345406s" Jul 07 17:54:14.423253 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:14.423213 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:14.423436 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:14.423352 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqq9h" podUID="3a1d71c1-6f70-441c-87bb-1f997df1fd5d" Jul 07 17:54:14.712718 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:14.712681 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2d819c_e495_47ec_a1e7_4b2582a195bd.slice/crio-4de45c4cc46a647980f3da43c2be2200eebe0283d3db3676133bfe5e6d72ff5c WatchSource:0}: Error finding container 4de45c4cc46a647980f3da43c2be2200eebe0283d3db3676133bfe5e6d72ff5c: Status 404 returned error can't find the container with id 4de45c4cc46a647980f3da43c2be2200eebe0283d3db3676133bfe5e6d72ff5c Jul 07 17:54:14.716127 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:14.716094 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ea083a_f7fb_4e82_a02a_b930204910b6.slice/crio-f1203c9f958ca6ae058c3d10cda752d93f08550163eef3b895120f5a1a27ab72 WatchSource:0}: Error finding container f1203c9f958ca6ae058c3d10cda752d93f08550163eef3b895120f5a1a27ab72: Status 404 returned error can't find the container with id f1203c9f958ca6ae058c3d10cda752d93f08550163eef3b895120f5a1a27ab72 Jul 07 17:54:14.717972 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:14.717850 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod797a1f8a_8d68_4c01_a38b_67d405653bd0.slice/crio-3f3d13fa9e72624ead0e9d99f3bc2164486786caf4cc10e373d5674b0e1fecfb WatchSource:0}: Error finding container 3f3d13fa9e72624ead0e9d99f3bc2164486786caf4cc10e373d5674b0e1fecfb: Status 404 returned error can't find the container with id 3f3d13fa9e72624ead0e9d99f3bc2164486786caf4cc10e373d5674b0e1fecfb Jul 07 17:54:14.720599 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:14.718974 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c3b9f0_ea6c_4262_b4e7_88945edc9f6f.slice/crio-f390b3902fa29d91e5e3be989b214731d550f0b20280d23be25cd62b5f25a62a WatchSource:0}: Error finding container f390b3902fa29d91e5e3be989b214731d550f0b20280d23be25cd62b5f25a62a: Status 404 returned error can't find the container with id f390b3902fa29d91e5e3be989b214731d550f0b20280d23be25cd62b5f25a62a Jul 07 17:54:14.723138 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:14.723091 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e0ef7c_fa8d_47da_9d76_bfe681f296d6.slice/crio-7b496592b1a213ff7d8cde8ef04654d41625724c53cd971985343fd7fcf37cee WatchSource:0}: Error finding container 7b496592b1a213ff7d8cde8ef04654d41625724c53cd971985343fd7fcf37cee: Status 404 returned error can't find the container with id 7b496592b1a213ff7d8cde8ef04654d41625724c53cd971985343fd7fcf37cee Jul 07 17:54:14.723747 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:14.723720 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1128d133_9305_4295_adec_94b196bd6f38.slice/crio-c07849ab799aa90189dde3ba212265f70ba156a819fddd453ea4db15da489002 WatchSource:0}: Error finding container c07849ab799aa90189dde3ba212265f70ba156a819fddd453ea4db15da489002: Status 404 returned error can't find the container with id c07849ab799aa90189dde3ba212265f70ba156a819fddd453ea4db15da489002 Jul 07 17:54:14.725724 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:14.725604 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod174cb3d6_9171_48b6_8e00_d094fa4b4757.slice/crio-f8a6744d60f3c659bb9259af9f5aed945ac4a9e3795b035a36dbd0ce08b869db WatchSource:0}: Error finding container f8a6744d60f3c659bb9259af9f5aed945ac4a9e3795b035a36dbd0ce08b869db: Status 404 returned error can't find the container with id f8a6744d60f3c659bb9259af9f5aed945ac4a9e3795b035a36dbd0ce08b869db Jul 07 17:54:14.916269 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:14.915700 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2fbb\" (UniqueName: \"kubernetes.io/projected/8a7f4631-8582-4636-832f-e026bb298b27-kube-api-access-g2fbb\") pod \"network-check-target-bmdgm\" (UID: \"8a7f4631-8582-4636-832f-e026bb298b27\") " pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:54:14.916269 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:14.916040 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:54:14.916269 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:14.916054 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a1d71c1-6f70-441c-87bb-1f997df1fd5d-original-pull-secret\") pod \"global-pull-secret-syncer-xqq9h\" (UID: \"3a1d71c1-6f70-441c-87bb-1f997df1fd5d\") " pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:14.916269 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:14.916071 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:54:14.916269 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:14.916090 2583 projected.go:196] Error preparing data for projected volume kube-api-access-g2fbb for pod openshift-network-diagnostics/network-check-target-bmdgm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:54:14.916269 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:14.916155 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a7f4631-8582-4636-832f-e026bb298b27-kube-api-access-g2fbb podName:8a7f4631-8582-4636-832f-e026bb298b27 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:16.916136155 +0000 UTC m=+5.998739859 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-g2fbb" (UniqueName: "kubernetes.io/projected/8a7f4631-8582-4636-832f-e026bb298b27-kube-api-access-g2fbb") pod "network-check-target-bmdgm" (UID: "8a7f4631-8582-4636-832f-e026bb298b27") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:54:14.916269 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:14.916177 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:54:14.916269 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:14.916223 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a1d71c1-6f70-441c-87bb-1f997df1fd5d-original-pull-secret podName:3a1d71c1-6f70-441c-87bb-1f997df1fd5d nodeName:}" failed. No retries permitted until 2026-07-07 17:54:16.916210499 +0000 UTC m=+5.998814183 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a1d71c1-6f70-441c-87bb-1f997df1fd5d-original-pull-secret") pod "global-pull-secret-syncer-xqq9h" (UID: "3a1d71c1-6f70-441c-87bb-1f997df1fd5d") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:54:15.017419 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:15.017378 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e10d7dd-5837-417c-bbe3-2cbf4ca1c144-metrics-certs\") pod \"network-metrics-daemon-lwjzj\" (UID: \"6e10d7dd-5837-417c-bbe3-2cbf4ca1c144\") " pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:54:15.017569 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:15.017528 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:54:15.017628 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:15.017597 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e10d7dd-5837-417c-bbe3-2cbf4ca1c144-metrics-certs podName:6e10d7dd-5837-417c-bbe3-2cbf4ca1c144 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:17.017581206 +0000 UTC m=+6.100184912 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e10d7dd-5837-417c-bbe3-2cbf4ca1c144-metrics-certs") pod "network-metrics-daemon-lwjzj" (UID: "6e10d7dd-5837-417c-bbe3-2cbf4ca1c144") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:54:15.338214 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:15.338095 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 17:49:13 +0000 UTC" deadline="2028-02-03 07:41:21.385760294 +0000 UTC" Jul 07 17:54:15.338214 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:15.338149 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13813h47m6.047621642s" Jul 07 17:54:15.425902 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:15.425855 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:54:15.426121 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:15.426028 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmdgm" podUID="8a7f4631-8582-4636-832f-e026bb298b27" Jul 07 17:54:15.426485 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:15.426462 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:54:15.426613 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:15.426588 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwjzj" podUID="6e10d7dd-5837-417c-bbe3-2cbf4ca1c144" Jul 07 17:54:15.441086 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:15.440577 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t2g29" event={"ID":"f09531d6-f2e3-47bb-acb2-de1092f4f9ff","Type":"ContainerStarted","Data":"53374cde170a01c8d251c981ee7a1148d7b80cf148984e7b2aa16a5f4dc11ff9"} Jul 07 17:54:15.447531 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:15.447490 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dm6hm" event={"ID":"d5ea083a-f7fb-4e82-a02a-b930204910b6","Type":"ContainerStarted","Data":"f1203c9f958ca6ae058c3d10cda752d93f08550163eef3b895120f5a1a27ab72"} Jul 07 17:54:15.454397 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:15.454351 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6zwkl" event={"ID":"f78b3750-ae77-4cf9-af3e-4ec59bc87dde","Type":"ContainerStarted","Data":"db081733acaddec1f852924c3e3a5935d5e49a60299e2aeafc82b948c040c100"} Jul 07 17:54:15.465126 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:15.464279 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-192.ec2.internal" event={"ID":"cf831b3419f5fd8b4190fd80bde7e0ae","Type":"ContainerStarted","Data":"eb6fe324e1053d2263e5fedfd7297e4512b1bf0a4a86198c55c333fae5b31d73"} Jul 07 17:54:15.475872 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:15.475707 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p4mnt" event={"ID":"e5c3b9f0-ea6c-4262-b4e7-88945edc9f6f","Type":"ContainerStarted","Data":"f390b3902fa29d91e5e3be989b214731d550f0b20280d23be25cd62b5f25a62a"} Jul 07 17:54:15.491827 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:15.491717 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-49qq2" event={"ID":"797a1f8a-8d68-4c01-a38b-67d405653bd0","Type":"ContainerStarted","Data":"3f3d13fa9e72624ead0e9d99f3bc2164486786caf4cc10e373d5674b0e1fecfb"} Jul 07 17:54:15.510123 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:15.510078 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" event={"ID":"5c2d819c-e495-47ec-a1e7-4b2582a195bd","Type":"ContainerStarted","Data":"4de45c4cc46a647980f3da43c2be2200eebe0283d3db3676133bfe5e6d72ff5c"} Jul 07 17:54:15.514342 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:15.514291 2583 generic.go:356] "Generic (PLEG): container finished" podID="b6ba5b519a9e1c4d8b4f40d858bd5c2f" containerID="516bb7146f76d32bd2ff6a4532ef67a4fa69dbee7a58e32429f00d4e03494da2" exitCode=0 Jul 07 17:54:15.514525 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:15.514414 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.ec2.internal" event={"ID":"b6ba5b519a9e1c4d8b4f40d858bd5c2f","Type":"ContainerDied","Data":"516bb7146f76d32bd2ff6a4532ef67a4fa69dbee7a58e32429f00d4e03494da2"} Jul 07 17:54:15.525163 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:15.524048 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z68qr" event={"ID":"174cb3d6-9171-48b6-8e00-d094fa4b4757","Type":"ContainerStarted","Data":"f8a6744d60f3c659bb9259af9f5aed945ac4a9e3795b035a36dbd0ce08b869db"} Jul 07 17:54:15.538582 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:15.538514 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-192.ec2.internal" podStartSLOduration=2.538490241 podStartE2EDuration="2.538490241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:54:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:54:15.484176043 +0000 UTC m=+4.566779748" watchObservedRunningTime="2026-07-07 17:54:15.538490241 +0000 UTC m=+4.621093947" Jul 07 17:54:15.545539 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:15.545482 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wfrk" event={"ID":"1128d133-9305-4295-adec-94b196bd6f38","Type":"ContainerStarted","Data":"c07849ab799aa90189dde3ba212265f70ba156a819fddd453ea4db15da489002"} Jul 07 17:54:15.555077 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:15.555031 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lbwbm" event={"ID":"a3e0ef7c-fa8d-47da-9d76-bfe681f296d6","Type":"ContainerStarted","Data":"7b496592b1a213ff7d8cde8ef04654d41625724c53cd971985343fd7fcf37cee"} Jul 07 17:54:16.424256 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:16.423678 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:16.424256 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:16.423830 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqq9h" podUID="3a1d71c1-6f70-441c-87bb-1f997df1fd5d" Jul 07 17:54:16.568491 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:16.568059 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.ec2.internal" event={"ID":"b6ba5b519a9e1c4d8b4f40d858bd5c2f","Type":"ContainerStarted","Data":"336836ca1a146167960b464ad2d2723f40d8e280d9fe16ebbf1e02498f519deb"} Jul 07 17:54:16.941366 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:16.940504 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2fbb\" (UniqueName: \"kubernetes.io/projected/8a7f4631-8582-4636-832f-e026bb298b27-kube-api-access-g2fbb\") pod \"network-check-target-bmdgm\" (UID: \"8a7f4631-8582-4636-832f-e026bb298b27\") " pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:54:16.941366 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:16.940597 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a1d71c1-6f70-441c-87bb-1f997df1fd5d-original-pull-secret\") pod \"global-pull-secret-syncer-xqq9h\" (UID: \"3a1d71c1-6f70-441c-87bb-1f997df1fd5d\") " pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:16.941366 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:16.940730 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:54:16.941366 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:16.940799 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a1d71c1-6f70-441c-87bb-1f997df1fd5d-original-pull-secret podName:3a1d71c1-6f70-441c-87bb-1f997df1fd5d nodeName:}" failed. No retries permitted until 2026-07-07 17:54:20.940780559 +0000 UTC m=+10.023384245 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a1d71c1-6f70-441c-87bb-1f997df1fd5d-original-pull-secret") pod "global-pull-secret-syncer-xqq9h" (UID: "3a1d71c1-6f70-441c-87bb-1f997df1fd5d") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:54:16.941366 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:16.940890 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:54:16.941366 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:16.940905 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:54:16.941366 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:16.940917 2583 projected.go:196] Error preparing data for projected volume kube-api-access-g2fbb for pod openshift-network-diagnostics/network-check-target-bmdgm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:54:16.941366 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:16.940979 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a7f4631-8582-4636-832f-e026bb298b27-kube-api-access-g2fbb podName:8a7f4631-8582-4636-832f-e026bb298b27 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:20.940941497 +0000 UTC m=+10.023545182 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-g2fbb" (UniqueName: "kubernetes.io/projected/8a7f4631-8582-4636-832f-e026bb298b27-kube-api-access-g2fbb") pod "network-check-target-bmdgm" (UID: "8a7f4631-8582-4636-832f-e026bb298b27") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:54:17.042189 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:17.042004 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e10d7dd-5837-417c-bbe3-2cbf4ca1c144-metrics-certs\") pod \"network-metrics-daemon-lwjzj\" (UID: \"6e10d7dd-5837-417c-bbe3-2cbf4ca1c144\") " pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:54:17.042397 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:17.042203 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:54:17.042397 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:17.042297 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e10d7dd-5837-417c-bbe3-2cbf4ca1c144-metrics-certs podName:6e10d7dd-5837-417c-bbe3-2cbf4ca1c144 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:21.042272414 +0000 UTC m=+10.124876116 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e10d7dd-5837-417c-bbe3-2cbf4ca1c144-metrics-certs") pod "network-metrics-daemon-lwjzj" (UID: "6e10d7dd-5837-417c-bbe3-2cbf4ca1c144") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:54:17.424680 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:17.424068 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:54:17.424680 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:17.424193 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwjzj" podUID="6e10d7dd-5837-417c-bbe3-2cbf4ca1c144" Jul 07 17:54:17.424680 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:17.424533 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:54:17.424680 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:17.424626 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmdgm" podUID="8a7f4631-8582-4636-832f-e026bb298b27" Jul 07 17:54:18.423585 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:18.423412 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:18.423585 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:18.423574 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqq9h" podUID="3a1d71c1-6f70-441c-87bb-1f997df1fd5d" Jul 07 17:54:19.424077 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:19.423219 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:54:19.424077 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:19.423384 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmdgm" podUID="8a7f4631-8582-4636-832f-e026bb298b27" Jul 07 17:54:19.424077 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:19.423835 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:54:19.424077 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:19.423978 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwjzj" podUID="6e10d7dd-5837-417c-bbe3-2cbf4ca1c144" Jul 07 17:54:20.423547 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:20.423472 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:20.423728 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:20.423652 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqq9h" podUID="3a1d71c1-6f70-441c-87bb-1f997df1fd5d" Jul 07 17:54:20.978242 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:20.978195 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2fbb\" (UniqueName: \"kubernetes.io/projected/8a7f4631-8582-4636-832f-e026bb298b27-kube-api-access-g2fbb\") pod \"network-check-target-bmdgm\" (UID: \"8a7f4631-8582-4636-832f-e026bb298b27\") " pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:54:20.978695 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:20.978285 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a1d71c1-6f70-441c-87bb-1f997df1fd5d-original-pull-secret\") pod \"global-pull-secret-syncer-xqq9h\" (UID: \"3a1d71c1-6f70-441c-87bb-1f997df1fd5d\") " pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:20.978695 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:20.978422 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:54:20.978695 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:20.978486 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a1d71c1-6f70-441c-87bb-1f997df1fd5d-original-pull-secret podName:3a1d71c1-6f70-441c-87bb-1f997df1fd5d nodeName:}" failed. No retries permitted until 2026-07-07 17:54:28.978466598 +0000 UTC m=+18.061070295 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a1d71c1-6f70-441c-87bb-1f997df1fd5d-original-pull-secret") pod "global-pull-secret-syncer-xqq9h" (UID: "3a1d71c1-6f70-441c-87bb-1f997df1fd5d") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:54:20.978974 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:20.978957 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:54:20.978974 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:20.978980 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:54:20.979109 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:20.978994 2583 projected.go:196] Error preparing data for projected volume kube-api-access-g2fbb for pod openshift-network-diagnostics/network-check-target-bmdgm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:54:20.979109 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:20.979039 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a7f4631-8582-4636-832f-e026bb298b27-kube-api-access-g2fbb podName:8a7f4631-8582-4636-832f-e026bb298b27 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:28.97902417 +0000 UTC m=+18.061627867 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-g2fbb" (UniqueName: "kubernetes.io/projected/8a7f4631-8582-4636-832f-e026bb298b27-kube-api-access-g2fbb") pod "network-check-target-bmdgm" (UID: "8a7f4631-8582-4636-832f-e026bb298b27") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:54:21.078825 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:21.078791 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e10d7dd-5837-417c-bbe3-2cbf4ca1c144-metrics-certs\") pod \"network-metrics-daemon-lwjzj\" (UID: \"6e10d7dd-5837-417c-bbe3-2cbf4ca1c144\") " pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:54:21.079020 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:21.078985 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:54:21.079086 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:21.079073 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e10d7dd-5837-417c-bbe3-2cbf4ca1c144-metrics-certs podName:6e10d7dd-5837-417c-bbe3-2cbf4ca1c144 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:29.079050496 +0000 UTC m=+18.161654181 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e10d7dd-5837-417c-bbe3-2cbf4ca1c144-metrics-certs") pod "network-metrics-daemon-lwjzj" (UID: "6e10d7dd-5837-417c-bbe3-2cbf4ca1c144") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:54:21.424823 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:21.424781 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:54:21.425032 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:21.424922 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmdgm" podUID="8a7f4631-8582-4636-832f-e026bb298b27" Jul 07 17:54:21.425452 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:21.425426 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:54:21.425616 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:21.425585 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwjzj" podUID="6e10d7dd-5837-417c-bbe3-2cbf4ca1c144" Jul 07 17:54:22.423373 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:22.423332 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:22.423858 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:22.423512 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqq9h" podUID="3a1d71c1-6f70-441c-87bb-1f997df1fd5d" Jul 07 17:54:23.423184 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:23.423155 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:54:23.423364 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:23.423198 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:54:23.423364 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:23.423299 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwjzj" podUID="6e10d7dd-5837-417c-bbe3-2cbf4ca1c144" Jul 07 17:54:23.423778 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:23.423666 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmdgm" podUID="8a7f4631-8582-4636-832f-e026bb298b27" Jul 07 17:54:24.423795 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:24.423765 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:24.424289 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:24.423920 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqq9h" podUID="3a1d71c1-6f70-441c-87bb-1f997df1fd5d" Jul 07 17:54:25.424083 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:25.424044 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:54:25.424635 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:25.424086 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:54:25.424635 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:25.424202 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwjzj" podUID="6e10d7dd-5837-417c-bbe3-2cbf4ca1c144" Jul 07 17:54:25.424635 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:25.424551 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmdgm" podUID="8a7f4631-8582-4636-832f-e026bb298b27" Jul 07 17:54:26.423284 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:26.423240 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:26.423494 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:26.423389 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqq9h" podUID="3a1d71c1-6f70-441c-87bb-1f997df1fd5d" Jul 07 17:54:27.423017 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:27.422978 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:54:27.423490 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:27.423107 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmdgm" podUID="8a7f4631-8582-4636-832f-e026bb298b27" Jul 07 17:54:27.423490 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:27.423165 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:54:27.423490 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:27.423272 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwjzj" podUID="6e10d7dd-5837-417c-bbe3-2cbf4ca1c144" Jul 07 17:54:28.422979 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:28.422944 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:28.423186 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:28.423065 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqq9h" podUID="3a1d71c1-6f70-441c-87bb-1f997df1fd5d" Jul 07 17:54:29.044247 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:29.044209 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a1d71c1-6f70-441c-87bb-1f997df1fd5d-original-pull-secret\") pod \"global-pull-secret-syncer-xqq9h\" (UID: \"3a1d71c1-6f70-441c-87bb-1f997df1fd5d\") " pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:29.044419 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:29.044267 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2fbb\" (UniqueName: \"kubernetes.io/projected/8a7f4631-8582-4636-832f-e026bb298b27-kube-api-access-g2fbb\") pod \"network-check-target-bmdgm\" (UID: \"8a7f4631-8582-4636-832f-e026bb298b27\") " pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:54:29.044419 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:29.044382 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:54:29.044419 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:29.044399 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:54:29.044419 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:29.044414 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:54:29.044597 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:29.044424 2583 projected.go:196] Error preparing data for projected volume kube-api-access-g2fbb for pod openshift-network-diagnostics/network-check-target-bmdgm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:54:29.044597 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:29.044454 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a1d71c1-6f70-441c-87bb-1f997df1fd5d-original-pull-secret podName:3a1d71c1-6f70-441c-87bb-1f997df1fd5d nodeName:}" failed. No retries permitted until 2026-07-07 17:54:45.044432845 +0000 UTC m=+34.127036527 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a1d71c1-6f70-441c-87bb-1f997df1fd5d-original-pull-secret") pod "global-pull-secret-syncer-xqq9h" (UID: "3a1d71c1-6f70-441c-87bb-1f997df1fd5d") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:54:29.044597 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:29.044476 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a7f4631-8582-4636-832f-e026bb298b27-kube-api-access-g2fbb podName:8a7f4631-8582-4636-832f-e026bb298b27 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:45.044460562 +0000 UTC m=+34.127064250 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-g2fbb" (UniqueName: "kubernetes.io/projected/8a7f4631-8582-4636-832f-e026bb298b27-kube-api-access-g2fbb") pod "network-check-target-bmdgm" (UID: "8a7f4631-8582-4636-832f-e026bb298b27") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:54:29.145143 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:29.145098 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e10d7dd-5837-417c-bbe3-2cbf4ca1c144-metrics-certs\") pod \"network-metrics-daemon-lwjzj\" (UID: \"6e10d7dd-5837-417c-bbe3-2cbf4ca1c144\") " pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:54:29.145309 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:29.145245 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:54:29.145410 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:29.145316 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e10d7dd-5837-417c-bbe3-2cbf4ca1c144-metrics-certs podName:6e10d7dd-5837-417c-bbe3-2cbf4ca1c144 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:45.145295442 +0000 UTC m=+34.227899128 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e10d7dd-5837-417c-bbe3-2cbf4ca1c144-metrics-certs") pod "network-metrics-daemon-lwjzj" (UID: "6e10d7dd-5837-417c-bbe3-2cbf4ca1c144") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:54:29.423208 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:29.423173 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:54:29.423649 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:29.423329 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwjzj" podUID="6e10d7dd-5837-417c-bbe3-2cbf4ca1c144" Jul 07 17:54:29.423649 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:29.423388 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:54:29.423649 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:29.423478 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmdgm" podUID="8a7f4631-8582-4636-832f-e026bb298b27" Jul 07 17:54:30.423634 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:30.423589 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:30.424075 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:30.423724 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqq9h" podUID="3a1d71c1-6f70-441c-87bb-1f997df1fd5d" Jul 07 17:54:31.424440 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:31.424407 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:54:31.424891 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:31.424506 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwjzj" podUID="6e10d7dd-5837-417c-bbe3-2cbf4ca1c144" Jul 07 17:54:31.424891 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:31.424545 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:54:31.424891 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:31.424595 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmdgm" podUID="8a7f4631-8582-4636-832f-e026bb298b27" Jul 07 17:54:32.423566 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:32.423529 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:32.423740 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:32.423654 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqq9h" podUID="3a1d71c1-6f70-441c-87bb-1f997df1fd5d" Jul 07 17:54:33.425124 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:33.425078 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:54:33.425770 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:33.425310 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwjzj" podUID="6e10d7dd-5837-417c-bbe3-2cbf4ca1c144" Jul 07 17:54:33.425770 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:33.425595 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:54:33.425770 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:33.425731 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmdgm" podUID="8a7f4631-8582-4636-832f-e026bb298b27" Jul 07 17:54:34.422960 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:34.422923 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:34.423145 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:34.423077 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqq9h" podUID="3a1d71c1-6f70-441c-87bb-1f997df1fd5d" Jul 07 17:54:34.625018 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:34.624752 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z68qr" event={"ID":"174cb3d6-9171-48b6-8e00-d094fa4b4757","Type":"ContainerStarted","Data":"c1e3317b9e9fdc6320f9f8a80ff012657bcde1991c6c474768fd1814dc0918ae"} Jul 07 17:54:34.626553 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:34.626521 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wfrk" event={"ID":"1128d133-9305-4295-adec-94b196bd6f38","Type":"ContainerStarted","Data":"0b58ad46224c37fae557ea0a7ad01c799c5af70f5581e309e37670c74f90ae5e"} Jul 07 17:54:34.628069 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:34.628037 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t2g29" event={"ID":"f09531d6-f2e3-47bb-acb2-de1092f4f9ff","Type":"ContainerStarted","Data":"9fa5dd193fdb1bbf3ba9e8107accac55cc24f17a234d5bd2f777acd908a387f8"} Jul 07 17:54:34.629491 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:34.629461 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dm6hm" event={"ID":"d5ea083a-f7fb-4e82-a02a-b930204910b6","Type":"ContainerStarted","Data":"bf0af1bca0d5de2262993328242e9092f34fe950d7aa547d30ea2f9507376c10"} Jul 07 17:54:34.630788 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:34.630761 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6zwkl" event={"ID":"f78b3750-ae77-4cf9-af3e-4ec59bc87dde","Type":"ContainerStarted","Data":"20f11060b899b2b82c3720cf89cd6bcb09a675dda3a7ab75a1b519975c1d66a1"} Jul 07 17:54:34.632097 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:34.632073 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p4mnt" event={"ID":"e5c3b9f0-ea6c-4262-b4e7-88945edc9f6f","Type":"ContainerStarted","Data":"d61731a5e1ad5aab3e8c82497b21b5902615108da565e0971b398e516296335d"} Jul 07 17:54:34.633328 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:34.633303 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-49qq2" event={"ID":"797a1f8a-8d68-4c01-a38b-67d405653bd0","Type":"ContainerStarted","Data":"6d396e629f1e7fee5ed53569c9efa1e2d3e98b1e02ef3ea678f77f09db4e9d70"} Jul 07 17:54:34.634649 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:34.634596 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" event={"ID":"5c2d819c-e495-47ec-a1e7-4b2582a195bd","Type":"ContainerStarted","Data":"5d523dfde76671a1e3f206bbcfaec6f85856159d369a25c42d620286b506c3b3"} Jul 07 17:54:34.653696 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:34.653632 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-192.ec2.internal" podStartSLOduration=21.653609999 podStartE2EDuration="21.653609999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:54:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:54:16.59042023 +0000 UTC m=+5.673023940" watchObservedRunningTime="2026-07-07 17:54:34.653609999 +0000 UTC m=+23.736213704" Jul 07 17:54:34.654451 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:34.654402 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-z68qr" podStartSLOduration=3.119255525 podStartE2EDuration="22.654389318s" totalImagesPullingTime="19.535133793s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:54:34.652320503 +0000 UTC m=+23.734924208" watchObservedRunningTime="2026-07-07 17:54:34.654389318 +0000 UTC m=+23.736993024" Jul 07 17:54:34.673132 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:34.673085 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-p4mnt" podStartSLOduration=3.551299549 podStartE2EDuration="22.673064132s" totalImagesPullingTime="19.121764583s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:54:34.670649623 +0000 UTC m=+23.753253329" watchObservedRunningTime="2026-07-07 17:54:34.673064132 +0000 UTC m=+23.755667837" Jul 07 17:54:34.717017 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:34.716926 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6zwkl" podStartSLOduration=8.029245797 podStartE2EDuration="22.716889021s" totalImagesPullingTime="14.687643224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:54:34.716231954 +0000 UTC m=+23.798835659" watchObservedRunningTime="2026-07-07 17:54:34.716889021 +0000 UTC m=+23.799492726" Jul 07 17:54:34.732099 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:34.731843 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dm6hm" podStartSLOduration=3.231914816 podStartE2EDuration="22.731825896s" totalImagesPullingTime="19.49991108s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:54:34.731593541 +0000 UTC m=+23.814197245" watchObservedRunningTime="2026-07-07 17:54:34.731825896 +0000 UTC m=+23.814429600" Jul 07 17:54:34.749392 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:34.749339 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-t2g29" podStartSLOduration=3.219793083 podStartE2EDuration="22.749322522s" totalImagesPullingTime="19.529529439s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:54:34.748731411 +0000 UTC m=+23.831335118" watchObservedRunningTime="2026-07-07 17:54:34.749322522 +0000 UTC m=+23.831926226" Jul 07 17:54:34.799588 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:34.799554 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:34.945560 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:34.945532 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:35.423597 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:35.423563 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:54:35.423782 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:35.423564 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:54:35.423782 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:35.423675 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwjzj" podUID="6e10d7dd-5837-417c-bbe3-2cbf4ca1c144" Jul 07 17:54:35.423782 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:35.423738 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmdgm" podUID="8a7f4631-8582-4636-832f-e026bb298b27" Jul 07 17:54:35.639024 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:35.638986 2583 generic.go:356] "Generic (PLEG): container finished" podID="797a1f8a-8d68-4c01-a38b-67d405653bd0" containerID="6d396e629f1e7fee5ed53569c9efa1e2d3e98b1e02ef3ea678f77f09db4e9d70" exitCode=0 Jul 07 17:54:35.639576 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:35.639083 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-49qq2" event={"ID":"797a1f8a-8d68-4c01-a38b-67d405653bd0","Type":"ContainerDied","Data":"6d396e629f1e7fee5ed53569c9efa1e2d3e98b1e02ef3ea678f77f09db4e9d70"} Jul 07 17:54:35.642841 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:35.642809 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zjzpw_5c2d819c-e495-47ec-a1e7-4b2582a195bd/ovn-acl-logging/0.log" Jul 07 17:54:35.643328 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:35.643297 2583 generic.go:356] "Generic (PLEG): container finished" podID="5c2d819c-e495-47ec-a1e7-4b2582a195bd" containerID="b92940984260c0a945b7a2b7e30f9e480362b76571233c9f649a2096d12fe526" exitCode=1 Jul 07 17:54:35.643526 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:35.643490 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" event={"ID":"5c2d819c-e495-47ec-a1e7-4b2582a195bd","Type":"ContainerStarted","Data":"6991e99d3f4ec24420fbaa73f8d919701848bed4f73cb6e24f8ea65d827c672c"} Jul 07 17:54:35.643602 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:35.643592 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" event={"ID":"5c2d819c-e495-47ec-a1e7-4b2582a195bd","Type":"ContainerStarted","Data":"041c99e22259d324292747fe3215436fde5c845375309234c6770bb55aac57eb"} Jul 07 17:54:35.643664 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:35.643652 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" event={"ID":"5c2d819c-e495-47ec-a1e7-4b2582a195bd","Type":"ContainerStarted","Data":"4e217513efb7b4728b38544bb02e5d675d4665309694d9662661cbd89db1ca88"} Jul 07 17:54:35.643720 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:35.643706 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" event={"ID":"5c2d819c-e495-47ec-a1e7-4b2582a195bd","Type":"ContainerStarted","Data":"9d794764995935fa47cc8e9326b707e1dda5dd5d6847484090a7e893bcf09699"} Jul 07 17:54:35.643773 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:35.643759 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" event={"ID":"5c2d819c-e495-47ec-a1e7-4b2582a195bd","Type":"ContainerDied","Data":"b92940984260c0a945b7a2b7e30f9e480362b76571233c9f649a2096d12fe526"} Jul 07 17:54:35.906568 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:35.906531 2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 07 17:54:36.357343 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:36.357209 2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T17:54:35.906565303Z","UUID":"01255aad-1c0d-4c13-8208-31f82667f3c1","Handler":null,"Name":"","Endpoint":""} Jul 07 17:54:36.359018 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:36.358998 2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 07 17:54:36.359205 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:36.359040 2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 07 17:54:36.423107 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:36.423073 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:36.423253 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:36.423179 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqq9h" podUID="3a1d71c1-6f70-441c-87bb-1f997df1fd5d" Jul 07 17:54:36.648416 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:36.648277 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lbwbm" event={"ID":"a3e0ef7c-fa8d-47da-9d76-bfe681f296d6","Type":"ContainerStarted","Data":"f9f2481496ce8c06d3c154bc4771ecc0c4214a0c1a164c3ab9830775ef4c357c"} Jul 07 17:54:36.650886 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:36.650855 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wfrk" event={"ID":"1128d133-9305-4295-adec-94b196bd6f38","Type":"ContainerStarted","Data":"703824ea447e44cf4e367efd9ecb4f1ae5dbfcd94081b6fde5e3a7ef106b01bc"} Jul 07 17:54:36.665956 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:36.665877 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lbwbm" podStartSLOduration=9.990426896 podStartE2EDuration="24.665853661s" totalImagesPullingTime="14.675426765s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:54:36.665620466 +0000 UTC m=+25.748224170" watchObservedRunningTime="2026-07-07 17:54:36.665853661 +0000 UTC m=+25.748457366" Jul 07 17:54:36.788345 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:36.788310 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:36.891942 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:36.891886 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:37.423569 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:37.423472 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:54:37.423569 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:37.423494 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:54:37.423767 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:37.423605 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwjzj" podUID="6e10d7dd-5837-417c-bbe3-2cbf4ca1c144" Jul 07 17:54:37.423767 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:37.423736 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmdgm" podUID="8a7f4631-8582-4636-832f-e026bb298b27" Jul 07 17:54:37.656151 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:37.656121 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zjzpw_5c2d819c-e495-47ec-a1e7-4b2582a195bd/ovn-acl-logging/0.log" Jul 07 17:54:37.656556 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:37.656480 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" event={"ID":"5c2d819c-e495-47ec-a1e7-4b2582a195bd","Type":"ContainerStarted","Data":"da22e1cbd3787853fc5367ea750665978d06bff6dfd3913b60215b801aa49fe3"} Jul 07 17:54:37.658576 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:37.658551 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wfrk" event={"ID":"1128d133-9305-4295-adec-94b196bd6f38","Type":"ContainerStarted","Data":"db387c372d8d69968e48733c397ea9a8ebd8e71c6b1d635b8f1bec2806b9a53b"} Jul 07 17:54:37.682279 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:37.682186 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wfrk" podStartSLOduration=4.033091615 podStartE2EDuration="25.682171211s" totalImagesPullingTime="21.649079596s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:54:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:54:37.681100989 +0000 UTC m=+26.763704693" watchObservedRunningTime="2026-07-07 17:54:37.682171211 +0000 UTC m=+26.764774914" Jul 07 17:54:37.961019 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:37.960919 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:38.059949 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:38.059893 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:38.423773 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:38.423741 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:38.423977 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:38.423863 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqq9h" podUID="3a1d71c1-6f70-441c-87bb-1f997df1fd5d" Jul 07 17:54:38.623410 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:38.623214 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:38.692595 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:38.692508 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:38.942977 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:38.942875 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6zwkl" Jul 07 17:54:38.943738 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:38.943715 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6zwkl" Jul 07 17:54:39.423410 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:39.423380 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:54:39.423583 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:39.423517 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwjzj" podUID="6e10d7dd-5837-417c-bbe3-2cbf4ca1c144" Jul 07 17:54:39.423948 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:39.423929 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:54:39.424038 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:39.424020 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmdgm" podUID="8a7f4631-8582-4636-832f-e026bb298b27" Jul 07 17:54:39.638707 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:39.638679 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:39.663715 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:39.663685 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6zwkl" Jul 07 17:54:39.664499 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:39.664480 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6zwkl" Jul 07 17:54:39.759608 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:39.759515 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:39.862229 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:39.862194 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:39.976892 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:39.976853 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:40.075975 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:40.075859 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:40.423665 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:40.423576 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:40.423953 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:40.423709 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqq9h" podUID="3a1d71c1-6f70-441c-87bb-1f997df1fd5d" Jul 07 17:54:40.669832 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:40.669802 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zjzpw_5c2d819c-e495-47ec-a1e7-4b2582a195bd/ovn-acl-logging/0.log" Jul 07 17:54:40.670210 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:40.670188 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" event={"ID":"5c2d819c-e495-47ec-a1e7-4b2582a195bd","Type":"ContainerStarted","Data":"9cfa6282d63c399fff5b84814f906dbf6607b3c49f541c0788cacea3c814086e"} Jul 07 17:54:40.670637 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:40.670621 2583 scope.go:117] "RemoveContainer" containerID="b92940984260c0a945b7a2b7e30f9e480362b76571233c9f649a2096d12fe526" Jul 07 17:54:41.425852 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:41.425675 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:54:41.426608 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:41.425739 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:54:41.426608 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:41.425933 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmdgm" podUID="8a7f4631-8582-4636-832f-e026bb298b27" Jul 07 17:54:41.426608 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:41.426018 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwjzj" podUID="6e10d7dd-5837-417c-bbe3-2cbf4ca1c144" Jul 07 17:54:41.674125 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:41.674091 2583 generic.go:356] "Generic (PLEG): container finished" podID="797a1f8a-8d68-4c01-a38b-67d405653bd0" containerID="ce13c324ebc187cae497feea36362f19216b204dbbe636a61374ca3e7f86d94a" exitCode=0 Jul 07 17:54:41.674281 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:41.674169 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-49qq2" event={"ID":"797a1f8a-8d68-4c01-a38b-67d405653bd0","Type":"ContainerDied","Data":"ce13c324ebc187cae497feea36362f19216b204dbbe636a61374ca3e7f86d94a"} Jul 07 17:54:41.678052 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:41.678033 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zjzpw_5c2d819c-e495-47ec-a1e7-4b2582a195bd/ovn-acl-logging/0.log" Jul 07 17:54:41.678476 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:41.678443 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" event={"ID":"5c2d819c-e495-47ec-a1e7-4b2582a195bd","Type":"ContainerStarted","Data":"6d1424164c7b953692000e8afa8b0a93df4dc1af2996f4d9c8e6fd8e8f10a627"} Jul 07 17:54:41.678717 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:41.678685 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:41.678812 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:41.678775 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:41.678858 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:41.678826 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:41.696819 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:41.696794 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:41.696988 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:41.696920 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:54:42.423622 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:42.423589 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:42.423780 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:42.423737 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqq9h" podUID="3a1d71c1-6f70-441c-87bb-1f997df1fd5d" Jul 07 17:54:42.610398 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:42.610343 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:42.647545 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:42.647418 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" podStartSLOduration=11.052468498 podStartE2EDuration="30.647392431s" totalImagesPullingTime="19.594923933s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:54:41.731625168 +0000 UTC m=+30.814228872" watchObservedRunningTime="2026-07-07 17:54:42.647392431 +0000 UTC m=+31.729996136" Jul 07 17:54:42.648309 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:42.648225 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bmdgm"] Jul 07 17:54:42.648443 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:42.648340 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:54:42.648443 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:42.648434 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmdgm" podUID="8a7f4631-8582-4636-832f-e026bb298b27" Jul 07 17:54:42.648831 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:42.648807 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xqq9h"] Jul 07 17:54:42.650005 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:42.649986 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lwjzj"] Jul 07 17:54:42.650095 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:42.650084 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:54:42.650209 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:42.650191 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwjzj" podUID="6e10d7dd-5837-417c-bbe3-2cbf4ca1c144" Jul 07 17:54:42.680408 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:42.680340 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:42.680541 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:42.680462 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqq9h" podUID="3a1d71c1-6f70-441c-87bb-1f997df1fd5d" Jul 07 17:54:43.685327 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:43.685293 2583 generic.go:356] "Generic (PLEG): container finished" podID="797a1f8a-8d68-4c01-a38b-67d405653bd0" containerID="ba9e6833ae0926321885fb2f819564f96168568ccc2dca2e1e331ca24dc7c5f8" exitCode=0 Jul 07 17:54:43.685686 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:43.685398 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-49qq2" event={"ID":"797a1f8a-8d68-4c01-a38b-67d405653bd0","Type":"ContainerDied","Data":"ba9e6833ae0926321885fb2f819564f96168568ccc2dca2e1e331ca24dc7c5f8"} Jul 07 17:54:44.423023 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:44.422810 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:44.423225 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:44.422810 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:54:44.423225 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:44.423118 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqq9h" podUID="3a1d71c1-6f70-441c-87bb-1f997df1fd5d" Jul 07 17:54:44.423225 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:44.422861 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:54:44.423364 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:44.423201 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmdgm" podUID="8a7f4631-8582-4636-832f-e026bb298b27" Jul 07 17:54:44.423413 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:44.423366 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwjzj" podUID="6e10d7dd-5837-417c-bbe3-2cbf4ca1c144" Jul 07 17:54:45.063758 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:45.063724 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2fbb\" (UniqueName: \"kubernetes.io/projected/8a7f4631-8582-4636-832f-e026bb298b27-kube-api-access-g2fbb\") pod \"network-check-target-bmdgm\" (UID: \"8a7f4631-8582-4636-832f-e026bb298b27\") " pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:54:45.064330 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:45.063780 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a1d71c1-6f70-441c-87bb-1f997df1fd5d-original-pull-secret\") pod \"global-pull-secret-syncer-xqq9h\" (UID: \"3a1d71c1-6f70-441c-87bb-1f997df1fd5d\") " pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:45.064330 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:45.063880 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 17:54:45.064330 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:45.063923 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 17:54:45.064330 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:45.063950 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 17:54:45.064330 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:45.063955 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a1d71c1-6f70-441c-87bb-1f997df1fd5d-original-pull-secret podName:3a1d71c1-6f70-441c-87bb-1f997df1fd5d nodeName:}" failed. No retries permitted until 2026-07-07 17:55:17.063937792 +0000 UTC m=+66.146541475 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a1d71c1-6f70-441c-87bb-1f997df1fd5d-original-pull-secret") pod "global-pull-secret-syncer-xqq9h" (UID: "3a1d71c1-6f70-441c-87bb-1f997df1fd5d") : object "kube-system"/"original-pull-secret" not registered Jul 07 17:54:45.064330 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:45.063964 2583 projected.go:196] Error preparing data for projected volume kube-api-access-g2fbb for pod openshift-network-diagnostics/network-check-target-bmdgm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:54:45.064330 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:45.064016 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a7f4631-8582-4636-832f-e026bb298b27-kube-api-access-g2fbb podName:8a7f4631-8582-4636-832f-e026bb298b27 nodeName:}" failed. No retries permitted until 2026-07-07 17:55:17.063998922 +0000 UTC m=+66.146602624 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-g2fbb" (UniqueName: "kubernetes.io/projected/8a7f4631-8582-4636-832f-e026bb298b27-kube-api-access-g2fbb") pod "network-check-target-bmdgm" (UID: "8a7f4631-8582-4636-832f-e026bb298b27") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 17:54:45.164218 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:45.164181 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e10d7dd-5837-417c-bbe3-2cbf4ca1c144-metrics-certs\") pod \"network-metrics-daemon-lwjzj\" (UID: \"6e10d7dd-5837-417c-bbe3-2cbf4ca1c144\") " pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:54:45.164426 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:45.164350 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:54:45.164426 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:45.164423 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e10d7dd-5837-417c-bbe3-2cbf4ca1c144-metrics-certs podName:6e10d7dd-5837-417c-bbe3-2cbf4ca1c144 nodeName:}" failed. No retries permitted until 2026-07-07 17:55:17.164402583 +0000 UTC m=+66.247006265 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e10d7dd-5837-417c-bbe3-2cbf4ca1c144-metrics-certs") pod "network-metrics-daemon-lwjzj" (UID: "6e10d7dd-5837-417c-bbe3-2cbf4ca1c144") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 17:54:45.692872 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:45.692838 2583 generic.go:356] "Generic (PLEG): container finished" podID="797a1f8a-8d68-4c01-a38b-67d405653bd0" containerID="4fcf2c78eacd7613f8865f4a680f565d4122dae1d2b5b3f775fcecbfd4c8e3db" exitCode=0 Jul 07 17:54:45.693051 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:45.692921 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-49qq2" event={"ID":"797a1f8a-8d68-4c01-a38b-67d405653bd0","Type":"ContainerDied","Data":"4fcf2c78eacd7613f8865f4a680f565d4122dae1d2b5b3f775fcecbfd4c8e3db"} Jul 07 17:54:46.422879 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:46.422843 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:54:46.422879 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:46.422879 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:46.423423 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:46.422979 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bmdgm" podUID="8a7f4631-8582-4636-832f-e026bb298b27" Jul 07 17:54:46.423423 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:46.423042 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:54:46.423423 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:46.423155 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xqq9h" podUID="3a1d71c1-6f70-441c-87bb-1f997df1fd5d" Jul 07 17:54:46.423423 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:46.423237 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwjzj" podUID="6e10d7dd-5837-417c-bbe3-2cbf4ca1c144" Jul 07 17:54:47.163484 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.163443 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:47.188465 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.188436 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-192.ec2.internal" event="NodeReady" Jul 07 17:54:47.188632 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.188601 2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 07 17:54:47.269029 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.268949 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-847hz"] Jul 07 17:54:47.273360 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.273320 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:47.292874 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.292836 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dr6hd"] Jul 07 17:54:47.293054 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.293007 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-847hz" Jul 07 17:54:47.296551 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.296222 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 07 17:54:47.296551 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.296417 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dtpjm\"" Jul 07 17:54:47.296735 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.296559 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 07 17:54:47.309076 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.309053 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-847hz"] Jul 07 17:54:47.309076 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.309086 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dr6hd"] Jul 07 17:54:47.309332 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.309170 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dr6hd" Jul 07 17:54:47.313391 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.313361 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-f6qmm\"" Jul 07 17:54:47.313540 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.313488 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 07 17:54:47.313776 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.313759 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 07 17:54:47.313850 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.313842 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 07 17:54:47.381532 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.381478 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e09f673-9441-4afe-a258-85b6b5203de8-metrics-tls\") pod \"dns-default-847hz\" (UID: \"4e09f673-9441-4afe-a258-85b6b5203de8\") " pod="openshift-dns/dns-default-847hz" Jul 07 17:54:47.381716 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.381546 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e09f673-9441-4afe-a258-85b6b5203de8-config-volume\") pod \"dns-default-847hz\" (UID: \"4e09f673-9441-4afe-a258-85b6b5203de8\") " pod="openshift-dns/dns-default-847hz" Jul 07 17:54:47.381716 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.381624 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbtmh\" (UniqueName: \"kubernetes.io/projected/4e09f673-9441-4afe-a258-85b6b5203de8-kube-api-access-vbtmh\") pod \"dns-default-847hz\" (UID: \"4e09f673-9441-4afe-a258-85b6b5203de8\") " pod="openshift-dns/dns-default-847hz" Jul 07 17:54:47.381716 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.381682 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e09f673-9441-4afe-a258-85b6b5203de8-tmp-dir\") pod \"dns-default-847hz\" (UID: \"4e09f673-9441-4afe-a258-85b6b5203de8\") " pod="openshift-dns/dns-default-847hz" Jul 07 17:54:47.482285 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.482246 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/686b9d0e-9891-4e46-b78b-c5030ceffa06-cert\") pod \"ingress-canary-dr6hd\" (UID: \"686b9d0e-9891-4e46-b78b-c5030ceffa06\") " pod="openshift-ingress-canary/ingress-canary-dr6hd" Jul 07 17:54:47.483017 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.482315 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbtmh\" (UniqueName: \"kubernetes.io/projected/4e09f673-9441-4afe-a258-85b6b5203de8-kube-api-access-vbtmh\") pod \"dns-default-847hz\" (UID: \"4e09f673-9441-4afe-a258-85b6b5203de8\") " pod="openshift-dns/dns-default-847hz" Jul 07 17:54:47.483017 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.482344 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-579vr\" (UniqueName: \"kubernetes.io/projected/686b9d0e-9891-4e46-b78b-c5030ceffa06-kube-api-access-579vr\") pod \"ingress-canary-dr6hd\" (UID: \"686b9d0e-9891-4e46-b78b-c5030ceffa06\") " pod="openshift-ingress-canary/ingress-canary-dr6hd" Jul 07 17:54:47.483017 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.482380 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e09f673-9441-4afe-a258-85b6b5203de8-tmp-dir\") pod \"dns-default-847hz\" (UID: \"4e09f673-9441-4afe-a258-85b6b5203de8\") " pod="openshift-dns/dns-default-847hz" Jul 07 17:54:47.483017 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.482398 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e09f673-9441-4afe-a258-85b6b5203de8-metrics-tls\") pod \"dns-default-847hz\" (UID: \"4e09f673-9441-4afe-a258-85b6b5203de8\") " pod="openshift-dns/dns-default-847hz" Jul 07 17:54:47.483017 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.482472 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e09f673-9441-4afe-a258-85b6b5203de8-config-volume\") pod \"dns-default-847hz\" (UID: \"4e09f673-9441-4afe-a258-85b6b5203de8\") " pod="openshift-dns/dns-default-847hz" Jul 07 17:54:47.483017 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:47.482497 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 17:54:47.483017 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:47.482557 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e09f673-9441-4afe-a258-85b6b5203de8-metrics-tls podName:4e09f673-9441-4afe-a258-85b6b5203de8 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:47.982538281 +0000 UTC m=+37.065141968 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e09f673-9441-4afe-a258-85b6b5203de8-metrics-tls") pod "dns-default-847hz" (UID: "4e09f673-9441-4afe-a258-85b6b5203de8") : secret "dns-default-metrics-tls" not found Jul 07 17:54:47.483017 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.482930 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e09f673-9441-4afe-a258-85b6b5203de8-tmp-dir\") pod \"dns-default-847hz\" (UID: \"4e09f673-9441-4afe-a258-85b6b5203de8\") " pod="openshift-dns/dns-default-847hz" Jul 07 17:54:47.483335 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.483040 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e09f673-9441-4afe-a258-85b6b5203de8-config-volume\") pod \"dns-default-847hz\" (UID: \"4e09f673-9441-4afe-a258-85b6b5203de8\") " pod="openshift-dns/dns-default-847hz" Jul 07 17:54:47.493132 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.493100 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:47.497165 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.497132 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbtmh\" (UniqueName: \"kubernetes.io/projected/4e09f673-9441-4afe-a258-85b6b5203de8-kube-api-access-vbtmh\") pod \"dns-default-847hz\" (UID: \"4e09f673-9441-4afe-a258-85b6b5203de8\") " pod="openshift-dns/dns-default-847hz" Jul 07 17:54:47.583834 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.583713 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/686b9d0e-9891-4e46-b78b-c5030ceffa06-cert\") pod \"ingress-canary-dr6hd\" (UID: \"686b9d0e-9891-4e46-b78b-c5030ceffa06\") " pod="openshift-ingress-canary/ingress-canary-dr6hd" Jul 07 17:54:47.583834 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.583772 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-579vr\" (UniqueName: \"kubernetes.io/projected/686b9d0e-9891-4e46-b78b-c5030ceffa06-kube-api-access-579vr\") pod \"ingress-canary-dr6hd\" (UID: \"686b9d0e-9891-4e46-b78b-c5030ceffa06\") " pod="openshift-ingress-canary/ingress-canary-dr6hd" Jul 07 17:54:47.586921 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.586879 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/686b9d0e-9891-4e46-b78b-c5030ceffa06-cert\") pod \"ingress-canary-dr6hd\" (UID: \"686b9d0e-9891-4e46-b78b-c5030ceffa06\") " pod="openshift-ingress-canary/ingress-canary-dr6hd" Jul 07 17:54:47.595508 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.595476 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:47.596562 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.596522 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-579vr\" (UniqueName: \"kubernetes.io/projected/686b9d0e-9891-4e46-b78b-c5030ceffa06-kube-api-access-579vr\") pod \"ingress-canary-dr6hd\" (UID: \"686b9d0e-9891-4e46-b78b-c5030ceffa06\") " pod="openshift-ingress-canary/ingress-canary-dr6hd" Jul 07 17:54:47.618545 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.618506 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dr6hd" Jul 07 17:54:47.832477 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.832443 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dr6hd"] Jul 07 17:54:47.839792 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:47.839637 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod686b9d0e_9891_4e46_b78b_c5030ceffa06.slice/crio-82d6b2adcdd32f282471f497711101d3b7aff8d049ccac884fc3e3769cf96e71 WatchSource:0}: Error finding container 82d6b2adcdd32f282471f497711101d3b7aff8d049ccac884fc3e3769cf96e71: Status 404 returned error can't find the container with id 82d6b2adcdd32f282471f497711101d3b7aff8d049ccac884fc3e3769cf96e71 Jul 07 17:54:47.986800 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:47.986752 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e09f673-9441-4afe-a258-85b6b5203de8-metrics-tls\") pod \"dns-default-847hz\" (UID: \"4e09f673-9441-4afe-a258-85b6b5203de8\") " pod="openshift-dns/dns-default-847hz" Jul 07 17:54:47.986986 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:47.986917 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 17:54:47.987035 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:47.986989 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e09f673-9441-4afe-a258-85b6b5203de8-metrics-tls podName:4e09f673-9441-4afe-a258-85b6b5203de8 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:48.986972838 +0000 UTC m=+38.069576524 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e09f673-9441-4afe-a258-85b6b5203de8-metrics-tls") pod "dns-default-847hz" (UID: "4e09f673-9441-4afe-a258-85b6b5203de8") : secret "dns-default-metrics-tls" not found Jul 07 17:54:48.423334 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:48.423287 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:54:48.423511 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:48.423305 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:54:48.423511 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:48.423470 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:54:48.428748 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:48.428716 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 17:54:48.429039 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:48.429016 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g4wsg\"" Jul 07 17:54:48.429131 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:48.429041 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 17:54:48.429474 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:48.429452 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 17:54:48.429566 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:48.429451 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 17:54:48.437338 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:48.437128 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n28xq\"" Jul 07 17:54:48.703802 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:48.703719 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dr6hd" event={"ID":"686b9d0e-9891-4e46-b78b-c5030ceffa06","Type":"ContainerStarted","Data":"82d6b2adcdd32f282471f497711101d3b7aff8d049ccac884fc3e3769cf96e71"} Jul 07 17:54:48.995612 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:48.995525 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e09f673-9441-4afe-a258-85b6b5203de8-metrics-tls\") pod \"dns-default-847hz\" (UID: \"4e09f673-9441-4afe-a258-85b6b5203de8\") " pod="openshift-dns/dns-default-847hz" Jul 07 17:54:48.995759 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:48.995715 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 17:54:48.995812 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:48.995801 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e09f673-9441-4afe-a258-85b6b5203de8-metrics-tls podName:4e09f673-9441-4afe-a258-85b6b5203de8 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:50.995777371 +0000 UTC m=+40.078381066 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e09f673-9441-4afe-a258-85b6b5203de8-metrics-tls") pod "dns-default-847hz" (UID: "4e09f673-9441-4afe-a258-85b6b5203de8") : secret "dns-default-metrics-tls" not found Jul 07 17:54:50.665016 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.664978 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ppncs"] Jul 07 17:54:50.678368 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.678339 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ppncs" Jul 07 17:54:50.681138 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.681109 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 07 17:54:50.681283 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.681109 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 07 17:54:50.682266 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.682225 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-mkxtg\"" Jul 07 17:54:50.682704 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.682682 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ppncs"] Jul 07 17:54:50.774830 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.774799 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bjv6p"] Jul 07 17:54:50.784210 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.784178 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bjv6p" Jul 07 17:54:50.787913 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.787426 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 07 17:54:50.787913 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.787530 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 07 17:54:50.787913 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.787689 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 07 17:54:50.788774 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.788590 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-g46fs\"" Jul 07 17:54:50.789120 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.789093 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6d876b7659-t4r6h"] Jul 07 17:54:50.789230 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.789178 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 07 17:54:50.798749 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.798723 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bjv6p"] Jul 07 17:54:50.798869 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.798835 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:54:50.802886 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.802865 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 17:54:50.803085 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.803065 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 07 17:54:50.803712 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.803684 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 17:54:50.803830 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.803713 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-rfvlw\"" Jul 07 17:54:50.803830 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.803693 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 07 17:54:50.803830 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.803691 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 07 17:54:50.811007 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.810982 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 07 17:54:50.812220 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.812200 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6chh\" (UniqueName: \"kubernetes.io/projected/2a31b62e-e3ad-4c70-a495-37a30122dc22-kube-api-access-w6chh\") pod \"volume-data-source-validator-5d456fdbcd-ppncs\" (UID: \"2a31b62e-e3ad-4c70-a495-37a30122dc22\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ppncs" Jul 07 17:54:50.823299 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.823264 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6d876b7659-t4r6h"] Jul 07 17:54:50.860708 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.860674 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:50.871248 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.871217 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-wqfm4"] Jul 07 17:54:50.886430 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.886399 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-644cfc977-5rqx8"] Jul 07 17:54:50.886606 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.886506 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" Jul 07 17:54:50.890699 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.890663 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 07 17:54:50.891125 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.891100 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 07 17:54:50.891272 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.891163 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-l6vsl\"" Jul 07 17:54:50.891437 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.891382 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 07 17:54:50.891566 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.891501 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 07 17:54:50.899992 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.899001 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:54:50.900138 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.900066 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 07 17:54:50.901009 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.900982 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-wqfm4"] Jul 07 17:54:50.903556 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.903528 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 07 17:54:50.903712 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.903695 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 07 17:54:50.903811 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.903796 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-c44kt\"" Jul 07 17:54:50.903985 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.903969 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 07 17:54:50.912873 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.912835 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2ed38cd7-9e63-4b1b-97cb-86748bb60a04-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-bjv6p\" (UID: \"2ed38cd7-9e63-4b1b-97cb-86748bb60a04\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bjv6p" Jul 07 17:54:50.913055 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.912930 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d9f9071-7f2b-4944-88fc-130da6a92ba5-metrics-certs\") pod \"router-default-6d876b7659-t4r6h\" (UID: \"4d9f9071-7f2b-4944-88fc-130da6a92ba5\") " pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:54:50.913055 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.912960 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5blth\" (UniqueName: \"kubernetes.io/projected/4d9f9071-7f2b-4944-88fc-130da6a92ba5-kube-api-access-5blth\") pod \"router-default-6d876b7659-t4r6h\" (UID: \"4d9f9071-7f2b-4944-88fc-130da6a92ba5\") " pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:54:50.913055 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.912986 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4d9f9071-7f2b-4944-88fc-130da6a92ba5-default-certificate\") pod \"router-default-6d876b7659-t4r6h\" (UID: \"4d9f9071-7f2b-4944-88fc-130da6a92ba5\") " pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:54:50.913055 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.913020 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkzf7\" (UniqueName: \"kubernetes.io/projected/2ed38cd7-9e63-4b1b-97cb-86748bb60a04-kube-api-access-wkzf7\") pod \"kube-storage-version-migrator-operator-846bf668f7-bjv6p\" (UID: \"2ed38cd7-9e63-4b1b-97cb-86748bb60a04\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bjv6p" Jul 07 17:54:50.913257 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.913068 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4d9f9071-7f2b-4944-88fc-130da6a92ba5-stats-auth\") pod \"router-default-6d876b7659-t4r6h\" (UID: \"4d9f9071-7f2b-4944-88fc-130da6a92ba5\") " pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:54:50.913257 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.913160 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2ed38cd7-9e63-4b1b-97cb-86748bb60a04-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-bjv6p\" (UID: \"2ed38cd7-9e63-4b1b-97cb-86748bb60a04\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bjv6p" Jul 07 17:54:50.913257 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.913195 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d9f9071-7f2b-4944-88fc-130da6a92ba5-service-ca-bundle\") pod \"router-default-6d876b7659-t4r6h\" (UID: \"4d9f9071-7f2b-4944-88fc-130da6a92ba5\") " pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:54:50.913257 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.913240 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6chh\" (UniqueName: \"kubernetes.io/projected/2a31b62e-e3ad-4c70-a495-37a30122dc22-kube-api-access-w6chh\") pod \"volume-data-source-validator-5d456fdbcd-ppncs\" (UID: \"2a31b62e-e3ad-4c70-a495-37a30122dc22\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ppncs" Jul 07 17:54:50.916669 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.916615 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 07 17:54:50.918886 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.918863 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-644cfc977-5rqx8"] Jul 07 17:54:50.925702 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.925671 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6chh\" (UniqueName: \"kubernetes.io/projected/2a31b62e-e3ad-4c70-a495-37a30122dc22-kube-api-access-w6chh\") pod \"volume-data-source-validator-5d456fdbcd-ppncs\" (UID: \"2a31b62e-e3ad-4c70-a495-37a30122dc22\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ppncs" Jul 07 17:54:50.965786 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.965752 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:50.990425 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:50.990392 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ppncs" Jul 07 17:54:51.014622 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.014577 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkzf7\" (UniqueName: \"kubernetes.io/projected/2ed38cd7-9e63-4b1b-97cb-86748bb60a04-kube-api-access-wkzf7\") pod \"kube-storage-version-migrator-operator-846bf668f7-bjv6p\" (UID: \"2ed38cd7-9e63-4b1b-97cb-86748bb60a04\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bjv6p" Jul 07 17:54:51.014800 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.014631 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4d9f9071-7f2b-4944-88fc-130da6a92ba5-stats-auth\") pod \"router-default-6d876b7659-t4r6h\" (UID: \"4d9f9071-7f2b-4944-88fc-130da6a92ba5\") " pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:54:51.014800 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.014663 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d9f9071-7f2b-4944-88fc-130da6a92ba5-metrics-certs\") pod \"router-default-6d876b7659-t4r6h\" (UID: \"4d9f9071-7f2b-4944-88fc-130da6a92ba5\") " pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:54:51.014800 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.014687 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5blth\" (UniqueName: \"kubernetes.io/projected/4d9f9071-7f2b-4944-88fc-130da6a92ba5-kube-api-access-5blth\") pod \"router-default-6d876b7659-t4r6h\" (UID: \"4d9f9071-7f2b-4944-88fc-130da6a92ba5\") " pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:54:51.014800 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.014713 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2ed38cd7-9e63-4b1b-97cb-86748bb60a04-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-bjv6p\" (UID: \"2ed38cd7-9e63-4b1b-97cb-86748bb60a04\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bjv6p" Jul 07 17:54:51.014800 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.014733 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4d9f9071-7f2b-4944-88fc-130da6a92ba5-default-certificate\") pod \"router-default-6d876b7659-t4r6h\" (UID: \"4d9f9071-7f2b-4944-88fc-130da6a92ba5\") " pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:54:51.014800 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.014778 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxplh\" (UniqueName: \"kubernetes.io/projected/997df85c-7632-47b5-a8a6-1c37f3346251-kube-api-access-cxplh\") pod \"console-operator-68b757fb67-wqfm4\" (UID: \"997df85c-7632-47b5-a8a6-1c37f3346251\") " pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" Jul 07 17:54:51.015116 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.014804 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13f33577-e89c-4c0f-9264-9a9971e8a17c-registry-certificates\") pod \"image-registry-644cfc977-5rqx8\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:54:51.015116 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.014822 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-bound-sa-token\") pod \"image-registry-644cfc977-5rqx8\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:54:51.015116 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.014840 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13f33577-e89c-4c0f-9264-9a9971e8a17c-ca-trust-extracted\") pod \"image-registry-644cfc977-5rqx8\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:54:51.015116 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.014860 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-registry-tls\") pod \"image-registry-644cfc977-5rqx8\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:54:51.015297 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:51.015136 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 17:54:51.015297 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:51.015215 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d9f9071-7f2b-4944-88fc-130da6a92ba5-metrics-certs podName:4d9f9071-7f2b-4944-88fc-130da6a92ba5 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:51.515193427 +0000 UTC m=+40.597797111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d9f9071-7f2b-4944-88fc-130da6a92ba5-metrics-certs") pod "router-default-6d876b7659-t4r6h" (UID: "4d9f9071-7f2b-4944-88fc-130da6a92ba5") : secret "router-metrics-certs-default" not found Jul 07 17:54:51.015419 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.015304 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/997df85c-7632-47b5-a8a6-1c37f3346251-trusted-ca\") pod \"console-operator-68b757fb67-wqfm4\" (UID: \"997df85c-7632-47b5-a8a6-1c37f3346251\") " pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" Jul 07 17:54:51.015419 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.015401 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/997df85c-7632-47b5-a8a6-1c37f3346251-config\") pod \"console-operator-68b757fb67-wqfm4\" (UID: \"997df85c-7632-47b5-a8a6-1c37f3346251\") " pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" Jul 07 17:54:51.015516 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.015432 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13f33577-e89c-4c0f-9264-9a9971e8a17c-installation-pull-secrets\") pod \"image-registry-644cfc977-5rqx8\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:54:51.015565 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.015507 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13f33577-e89c-4c0f-9264-9a9971e8a17c-image-registry-private-configuration\") pod \"image-registry-644cfc977-5rqx8\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:54:51.015565 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.015550 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13f33577-e89c-4c0f-9264-9a9971e8a17c-trusted-ca\") pod \"image-registry-644cfc977-5rqx8\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:54:51.015651 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.015586 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e09f673-9441-4afe-a258-85b6b5203de8-metrics-tls\") pod \"dns-default-847hz\" (UID: \"4e09f673-9441-4afe-a258-85b6b5203de8\") " pod="openshift-dns/dns-default-847hz" Jul 07 17:54:51.015651 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.015617 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2ed38cd7-9e63-4b1b-97cb-86748bb60a04-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-bjv6p\" (UID: \"2ed38cd7-9e63-4b1b-97cb-86748bb60a04\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bjv6p" Jul 07 17:54:51.015651 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.015643 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d9f9071-7f2b-4944-88fc-130da6a92ba5-service-ca-bundle\") pod \"router-default-6d876b7659-t4r6h\" (UID: \"4d9f9071-7f2b-4944-88fc-130da6a92ba5\") " pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:54:51.015779 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.015667 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/997df85c-7632-47b5-a8a6-1c37f3346251-serving-cert\") pod \"console-operator-68b757fb67-wqfm4\" (UID: \"997df85c-7632-47b5-a8a6-1c37f3346251\") " pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" Jul 07 17:54:51.015779 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:51.015681 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 17:54:51.015779 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.015680 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2ed38cd7-9e63-4b1b-97cb-86748bb60a04-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-bjv6p\" (UID: \"2ed38cd7-9e63-4b1b-97cb-86748bb60a04\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bjv6p" Jul 07 17:54:51.015779 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.015701 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87kzf\" (UniqueName: \"kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-kube-api-access-87kzf\") pod \"image-registry-644cfc977-5rqx8\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:54:51.015779 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:51.015723 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e09f673-9441-4afe-a258-85b6b5203de8-metrics-tls podName:4e09f673-9441-4afe-a258-85b6b5203de8 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:55.015709912 +0000 UTC m=+44.098313619 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e09f673-9441-4afe-a258-85b6b5203de8-metrics-tls") pod "dns-default-847hz" (UID: "4e09f673-9441-4afe-a258-85b6b5203de8") : secret "dns-default-metrics-tls" not found Jul 07 17:54:51.015989 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:51.015822 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d9f9071-7f2b-4944-88fc-130da6a92ba5-service-ca-bundle podName:4d9f9071-7f2b-4944-88fc-130da6a92ba5 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:51.515801917 +0000 UTC m=+40.598405625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4d9f9071-7f2b-4944-88fc-130da6a92ba5-service-ca-bundle") pod "router-default-6d876b7659-t4r6h" (UID: "4d9f9071-7f2b-4944-88fc-130da6a92ba5") : configmap references non-existent config key: service-ca.crt Jul 07 17:54:51.021930 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.017835 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4d9f9071-7f2b-4944-88fc-130da6a92ba5-stats-auth\") pod \"router-default-6d876b7659-t4r6h\" (UID: \"4d9f9071-7f2b-4944-88fc-130da6a92ba5\") " pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:54:51.021930 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.018035 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4d9f9071-7f2b-4944-88fc-130da6a92ba5-default-certificate\") pod \"router-default-6d876b7659-t4r6h\" (UID: \"4d9f9071-7f2b-4944-88fc-130da6a92ba5\") " pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:54:51.021930 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.018055 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2ed38cd7-9e63-4b1b-97cb-86748bb60a04-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-bjv6p\" (UID: \"2ed38cd7-9e63-4b1b-97cb-86748bb60a04\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bjv6p" Jul 07 17:54:51.026126 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.026070 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5blth\" (UniqueName: \"kubernetes.io/projected/4d9f9071-7f2b-4944-88fc-130da6a92ba5-kube-api-access-5blth\") pod \"router-default-6d876b7659-t4r6h\" (UID: \"4d9f9071-7f2b-4944-88fc-130da6a92ba5\") " pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:54:51.028390 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.028366 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkzf7\" (UniqueName: \"kubernetes.io/projected/2ed38cd7-9e63-4b1b-97cb-86748bb60a04-kube-api-access-wkzf7\") pod \"kube-storage-version-migrator-operator-846bf668f7-bjv6p\" (UID: \"2ed38cd7-9e63-4b1b-97cb-86748bb60a04\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bjv6p" Jul 07 17:54:51.095748 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.095710 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bjv6p" Jul 07 17:54:51.105261 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.105229 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:51.117072 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.117043 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13f33577-e89c-4c0f-9264-9a9971e8a17c-ca-trust-extracted\") pod \"image-registry-644cfc977-5rqx8\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:54:51.117219 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.117080 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-registry-tls\") pod \"image-registry-644cfc977-5rqx8\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:54:51.117219 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.117125 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/997df85c-7632-47b5-a8a6-1c37f3346251-trusted-ca\") pod \"console-operator-68b757fb67-wqfm4\" (UID: \"997df85c-7632-47b5-a8a6-1c37f3346251\") " pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" Jul 07 17:54:51.117219 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.117163 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/997df85c-7632-47b5-a8a6-1c37f3346251-config\") pod \"console-operator-68b757fb67-wqfm4\" (UID: \"997df85c-7632-47b5-a8a6-1c37f3346251\") " pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" Jul 07 17:54:51.117219 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:51.117183 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 17:54:51.117219 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:51.117203 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-644cfc977-5rqx8: secret "image-registry-tls" not found Jul 07 17:54:51.117445 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:51.117265 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-registry-tls podName:13f33577-e89c-4c0f-9264-9a9971e8a17c nodeName:}" failed. No retries permitted until 2026-07-07 17:54:51.617244233 +0000 UTC m=+40.699847938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-registry-tls") pod "image-registry-644cfc977-5rqx8" (UID: "13f33577-e89c-4c0f-9264-9a9971e8a17c") : secret "image-registry-tls" not found Jul 07 17:54:51.117445 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.117435 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13f33577-e89c-4c0f-9264-9a9971e8a17c-ca-trust-extracted\") pod \"image-registry-644cfc977-5rqx8\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:54:51.117556 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.117182 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13f33577-e89c-4c0f-9264-9a9971e8a17c-installation-pull-secrets\") pod \"image-registry-644cfc977-5rqx8\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:54:51.117622 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.117601 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13f33577-e89c-4c0f-9264-9a9971e8a17c-image-registry-private-configuration\") pod \"image-registry-644cfc977-5rqx8\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:54:51.117674 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.117641 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13f33577-e89c-4c0f-9264-9a9971e8a17c-trusted-ca\") pod \"image-registry-644cfc977-5rqx8\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:54:51.117725 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.117693 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/997df85c-7632-47b5-a8a6-1c37f3346251-serving-cert\") pod \"console-operator-68b757fb67-wqfm4\" (UID: \"997df85c-7632-47b5-a8a6-1c37f3346251\") " pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" Jul 07 17:54:51.117772 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.117729 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87kzf\" (UniqueName: \"kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-kube-api-access-87kzf\") pod \"image-registry-644cfc977-5rqx8\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:54:51.117836 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.117816 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxplh\" (UniqueName: \"kubernetes.io/projected/997df85c-7632-47b5-a8a6-1c37f3346251-kube-api-access-cxplh\") pod \"console-operator-68b757fb67-wqfm4\" (UID: \"997df85c-7632-47b5-a8a6-1c37f3346251\") " pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" Jul 07 17:54:51.118012 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.117869 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13f33577-e89c-4c0f-9264-9a9971e8a17c-registry-certificates\") pod \"image-registry-644cfc977-5rqx8\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:54:51.118012 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.117921 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-bound-sa-token\") pod \"image-registry-644cfc977-5rqx8\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:54:51.118172 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.118083 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/997df85c-7632-47b5-a8a6-1c37f3346251-config\") pod \"console-operator-68b757fb67-wqfm4\" (UID: \"997df85c-7632-47b5-a8a6-1c37f3346251\") " pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" Jul 07 17:54:51.118172 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.118158 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/997df85c-7632-47b5-a8a6-1c37f3346251-trusted-ca\") pod \"console-operator-68b757fb67-wqfm4\" (UID: \"997df85c-7632-47b5-a8a6-1c37f3346251\") " pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" Jul 07 17:54:51.118863 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.118836 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13f33577-e89c-4c0f-9264-9a9971e8a17c-registry-certificates\") pod \"image-registry-644cfc977-5rqx8\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:54:51.119063 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.119030 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13f33577-e89c-4c0f-9264-9a9971e8a17c-trusted-ca\") pod \"image-registry-644cfc977-5rqx8\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:54:51.120206 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.120180 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13f33577-e89c-4c0f-9264-9a9971e8a17c-installation-pull-secrets\") pod \"image-registry-644cfc977-5rqx8\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:54:51.121059 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.121030 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13f33577-e89c-4c0f-9264-9a9971e8a17c-image-registry-private-configuration\") pod \"image-registry-644cfc977-5rqx8\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:54:51.121396 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.121374 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/997df85c-7632-47b5-a8a6-1c37f3346251-serving-cert\") pod \"console-operator-68b757fb67-wqfm4\" (UID: \"997df85c-7632-47b5-a8a6-1c37f3346251\") " pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" Jul 07 17:54:51.131688 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.131653 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87kzf\" (UniqueName: \"kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-kube-api-access-87kzf\") pod \"image-registry-644cfc977-5rqx8\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:54:51.131872 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.131856 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-bound-sa-token\") pod \"image-registry-644cfc977-5rqx8\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:54:51.138525 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.138493 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxplh\" (UniqueName: \"kubernetes.io/projected/997df85c-7632-47b5-a8a6-1c37f3346251-kube-api-access-cxplh\") pod \"console-operator-68b757fb67-wqfm4\" (UID: \"997df85c-7632-47b5-a8a6-1c37f3346251\") " pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" Jul 07 17:54:51.202770 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.202673 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" Jul 07 17:54:51.207221 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.207173 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:51.522350 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.522251 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d9f9071-7f2b-4944-88fc-130da6a92ba5-service-ca-bundle\") pod \"router-default-6d876b7659-t4r6h\" (UID: \"4d9f9071-7f2b-4944-88fc-130da6a92ba5\") " pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:54:51.522350 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.522320 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d9f9071-7f2b-4944-88fc-130da6a92ba5-metrics-certs\") pod \"router-default-6d876b7659-t4r6h\" (UID: \"4d9f9071-7f2b-4944-88fc-130da6a92ba5\") " pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:54:51.522597 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:51.522444 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d9f9071-7f2b-4944-88fc-130da6a92ba5-service-ca-bundle podName:4d9f9071-7f2b-4944-88fc-130da6a92ba5 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:52.522423888 +0000 UTC m=+41.605027573 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4d9f9071-7f2b-4944-88fc-130da6a92ba5-service-ca-bundle") pod "router-default-6d876b7659-t4r6h" (UID: "4d9f9071-7f2b-4944-88fc-130da6a92ba5") : configmap references non-existent config key: service-ca.crt Jul 07 17:54:51.522597 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:51.522446 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 17:54:51.522597 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:51.522539 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d9f9071-7f2b-4944-88fc-130da6a92ba5-metrics-certs podName:4d9f9071-7f2b-4944-88fc-130da6a92ba5 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:52.522518064 +0000 UTC m=+41.605121763 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d9f9071-7f2b-4944-88fc-130da6a92ba5-metrics-certs") pod "router-default-6d876b7659-t4r6h" (UID: "4d9f9071-7f2b-4944-88fc-130da6a92ba5") : secret "router-metrics-certs-default" not found Jul 07 17:54:51.623286 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:51.623241 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-registry-tls\") pod \"image-registry-644cfc977-5rqx8\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:54:51.623470 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:51.623399 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 17:54:51.623470 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:51.623427 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-644cfc977-5rqx8: secret "image-registry-tls" not found Jul 07 17:54:51.623585 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:51.623501 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-registry-tls podName:13f33577-e89c-4c0f-9264-9a9971e8a17c nodeName:}" failed. No retries permitted until 2026-07-07 17:54:52.623477686 +0000 UTC m=+41.706081385 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-registry-tls") pod "image-registry-644cfc977-5rqx8" (UID: "13f33577-e89c-4c0f-9264-9a9971e8a17c") : secret "image-registry-tls" not found Jul 07 17:54:52.531349 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:52.530505 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d9f9071-7f2b-4944-88fc-130da6a92ba5-service-ca-bundle\") pod \"router-default-6d876b7659-t4r6h\" (UID: \"4d9f9071-7f2b-4944-88fc-130da6a92ba5\") " pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:54:52.531349 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:52.530589 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d9f9071-7f2b-4944-88fc-130da6a92ba5-metrics-certs\") pod \"router-default-6d876b7659-t4r6h\" (UID: \"4d9f9071-7f2b-4944-88fc-130da6a92ba5\") " pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:54:52.531349 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:52.530752 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 17:54:52.531349 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:52.530817 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d9f9071-7f2b-4944-88fc-130da6a92ba5-metrics-certs podName:4d9f9071-7f2b-4944-88fc-130da6a92ba5 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:54.530797666 +0000 UTC m=+43.613401369 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d9f9071-7f2b-4944-88fc-130da6a92ba5-metrics-certs") pod "router-default-6d876b7659-t4r6h" (UID: "4d9f9071-7f2b-4944-88fc-130da6a92ba5") : secret "router-metrics-certs-default" not found Jul 07 17:54:52.531349 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:52.531315 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d9f9071-7f2b-4944-88fc-130da6a92ba5-service-ca-bundle podName:4d9f9071-7f2b-4944-88fc-130da6a92ba5 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:54.531297468 +0000 UTC m=+43.613901150 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4d9f9071-7f2b-4944-88fc-130da6a92ba5-service-ca-bundle") pod "router-default-6d876b7659-t4r6h" (UID: "4d9f9071-7f2b-4944-88fc-130da6a92ba5") : configmap references non-existent config key: service-ca.crt Jul 07 17:54:52.631740 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:52.631285 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-registry-tls\") pod \"image-registry-644cfc977-5rqx8\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:54:52.631869 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:52.631747 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 17:54:52.631869 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:52.631766 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-644cfc977-5rqx8: secret "image-registry-tls" not found Jul 07 17:54:52.631869 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:52.631830 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-registry-tls podName:13f33577-e89c-4c0f-9264-9a9971e8a17c nodeName:}" failed. No retries permitted until 2026-07-07 17:54:54.631809279 +0000 UTC m=+43.714412964 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-registry-tls") pod "image-registry-644cfc977-5rqx8" (UID: "13f33577-e89c-4c0f-9264-9a9971e8a17c") : secret "image-registry-tls" not found Jul 07 17:54:52.675797 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:52.675771 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-wqfm4"] Jul 07 17:54:52.679350 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:52.679324 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod997df85c_7632_47b5_a8a6_1c37f3346251.slice/crio-b05dc4f241e343644b3d6b7185a72fb0d4e68a886ae73a4fdfc0e08a690d749b WatchSource:0}: Error finding container b05dc4f241e343644b3d6b7185a72fb0d4e68a886ae73a4fdfc0e08a690d749b: Status 404 returned error can't find the container with id b05dc4f241e343644b3d6b7185a72fb0d4e68a886ae73a4fdfc0e08a690d749b Jul 07 17:54:52.687924 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:52.687880 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bjv6p"] Jul 07 17:54:52.690081 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:52.690059 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed38cd7_9e63_4b1b_97cb_86748bb60a04.slice/crio-9e90bee4a07baee0fba2ea98fa854995881959900c3db285176b93525a990ea1 WatchSource:0}: Error finding container 9e90bee4a07baee0fba2ea98fa854995881959900c3db285176b93525a990ea1: Status 404 returned error can't find the container with id 9e90bee4a07baee0fba2ea98fa854995881959900c3db285176b93525a990ea1 Jul 07 17:54:52.693037 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:52.693002 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ppncs"] Jul 07 17:54:52.697035 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:54:52.697005 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a31b62e_e3ad_4c70_a495_37a30122dc22.slice/crio-dcbaff2255ec12f13d1d4685c868b9fb0c17a0c26e6b3fe2154cc7455aacff5d WatchSource:0}: Error finding container dcbaff2255ec12f13d1d4685c868b9fb0c17a0c26e6b3fe2154cc7455aacff5d: Status 404 returned error can't find the container with id dcbaff2255ec12f13d1d4685c868b9fb0c17a0c26e6b3fe2154cc7455aacff5d Jul 07 17:54:52.717792 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:52.717592 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-49qq2" event={"ID":"797a1f8a-8d68-4c01-a38b-67d405653bd0","Type":"ContainerStarted","Data":"2d6004e60c766478b97d863c7ef2c675e9ee61bc901ef29e23f80fad60b8b82a"} Jul 07 17:54:52.719211 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:52.719181 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" event={"ID":"997df85c-7632-47b5-a8a6-1c37f3346251","Type":"ContainerStarted","Data":"b05dc4f241e343644b3d6b7185a72fb0d4e68a886ae73a4fdfc0e08a690d749b"} Jul 07 17:54:52.720761 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:52.720733 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ppncs" event={"ID":"2a31b62e-e3ad-4c70-a495-37a30122dc22","Type":"ContainerStarted","Data":"dcbaff2255ec12f13d1d4685c868b9fb0c17a0c26e6b3fe2154cc7455aacff5d"} Jul 07 17:54:52.722629 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:52.722595 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dr6hd" event={"ID":"686b9d0e-9891-4e46-b78b-c5030ceffa06","Type":"ContainerStarted","Data":"6522c61a4551c928db144bb526e09e12ac931898b98c7a5b6ba7cb5993e363c8"} Jul 07 17:54:52.724079 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:52.724057 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bjv6p" event={"ID":"2ed38cd7-9e63-4b1b-97cb-86748bb60a04","Type":"ContainerStarted","Data":"9e90bee4a07baee0fba2ea98fa854995881959900c3db285176b93525a990ea1"} Jul 07 17:54:52.768238 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:52.768182 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dr6hd" podStartSLOduration=1.111036053 podStartE2EDuration="5.768163687s" totalImagesPullingTime="4.657127634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:54:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:54:52.767112505 +0000 UTC m=+41.849716211" watchObservedRunningTime="2026-07-07 17:54:52.768163687 +0000 UTC m=+41.850767381" Jul 07 17:54:52.855713 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:52.855678 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:52.982170 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:52.982082 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:53.733744 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:53.733219 2583 generic.go:356] "Generic (PLEG): container finished" podID="797a1f8a-8d68-4c01-a38b-67d405653bd0" containerID="2d6004e60c766478b97d863c7ef2c675e9ee61bc901ef29e23f80fad60b8b82a" exitCode=0 Jul 07 17:54:53.733744 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:53.733317 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-49qq2" event={"ID":"797a1f8a-8d68-4c01-a38b-67d405653bd0","Type":"ContainerDied","Data":"2d6004e60c766478b97d863c7ef2c675e9ee61bc901ef29e23f80fad60b8b82a"} Jul 07 17:54:54.547508 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:54.547454 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d9f9071-7f2b-4944-88fc-130da6a92ba5-service-ca-bundle\") pod \"router-default-6d876b7659-t4r6h\" (UID: \"4d9f9071-7f2b-4944-88fc-130da6a92ba5\") " pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:54:54.547701 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:54.547527 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d9f9071-7f2b-4944-88fc-130da6a92ba5-metrics-certs\") pod \"router-default-6d876b7659-t4r6h\" (UID: \"4d9f9071-7f2b-4944-88fc-130da6a92ba5\") " pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:54:54.547701 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:54.547641 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d9f9071-7f2b-4944-88fc-130da6a92ba5-service-ca-bundle podName:4d9f9071-7f2b-4944-88fc-130da6a92ba5 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:58.547618615 +0000 UTC m=+47.630222296 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4d9f9071-7f2b-4944-88fc-130da6a92ba5-service-ca-bundle") pod "router-default-6d876b7659-t4r6h" (UID: "4d9f9071-7f2b-4944-88fc-130da6a92ba5") : configmap references non-existent config key: service-ca.crt Jul 07 17:54:54.547833 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:54.547724 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 17:54:54.547833 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:54.547807 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d9f9071-7f2b-4944-88fc-130da6a92ba5-metrics-certs podName:4d9f9071-7f2b-4944-88fc-130da6a92ba5 nodeName:}" failed. No retries permitted until 2026-07-07 17:54:58.547789928 +0000 UTC m=+47.630393632 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d9f9071-7f2b-4944-88fc-130da6a92ba5-metrics-certs") pod "router-default-6d876b7659-t4r6h" (UID: "4d9f9071-7f2b-4944-88fc-130da6a92ba5") : secret "router-metrics-certs-default" not found Jul 07 17:54:54.648022 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:54.647988 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-registry-tls\") pod \"image-registry-644cfc977-5rqx8\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:54:54.648193 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:54.648164 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 17:54:54.648193 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:54.648188 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-644cfc977-5rqx8: secret "image-registry-tls" not found Jul 07 17:54:54.648275 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:54.648260 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-registry-tls podName:13f33577-e89c-4c0f-9264-9a9971e8a17c nodeName:}" failed. No retries permitted until 2026-07-07 17:54:58.648237936 +0000 UTC m=+47.730841618 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-registry-tls") pod "image-registry-644cfc977-5rqx8" (UID: "13f33577-e89c-4c0f-9264-9a9971e8a17c") : secret "image-registry-tls" not found Jul 07 17:54:54.740724 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:54.740683 2583 generic.go:356] "Generic (PLEG): container finished" podID="797a1f8a-8d68-4c01-a38b-67d405653bd0" containerID="67ccd0da8922839dfe108053a23b429007ae8aecd0eeb1aef90ca7868a543036" exitCode=0 Jul 07 17:54:54.741182 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:54.740771 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-49qq2" event={"ID":"797a1f8a-8d68-4c01-a38b-67d405653bd0","Type":"ContainerDied","Data":"67ccd0da8922839dfe108053a23b429007ae8aecd0eeb1aef90ca7868a543036"} Jul 07 17:54:54.742826 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:54.742799 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ppncs" event={"ID":"2a31b62e-e3ad-4c70-a495-37a30122dc22","Type":"ContainerStarted","Data":"ffd0d05750d3e12f8249b81e309c795f333141f7c6782319b6381e0787c37be0"} Jul 07 17:54:54.796715 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:54.796632 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ppncs" podStartSLOduration=3.620464965 podStartE2EDuration="4.79660817s" totalImagesPullingTime="1.176143205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:54:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:54:54.796085042 +0000 UTC m=+43.878688746" watchObservedRunningTime="2026-07-07 17:54:54.79660817 +0000 UTC m=+43.879211875" Jul 07 17:54:54.902314 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:54.902276 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:55.051365 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:55.051327 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e09f673-9441-4afe-a258-85b6b5203de8-metrics-tls\") pod \"dns-default-847hz\" (UID: \"4e09f673-9441-4afe-a258-85b6b5203de8\") " pod="openshift-dns/dns-default-847hz" Jul 07 17:54:55.051563 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:55.051484 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 17:54:55.051563 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:55.051555 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e09f673-9441-4afe-a258-85b6b5203de8-metrics-tls podName:4e09f673-9441-4afe-a258-85b6b5203de8 nodeName:}" failed. No retries permitted until 2026-07-07 17:55:03.051536761 +0000 UTC m=+52.134140443 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e09f673-9441-4afe-a258-85b6b5203de8-metrics-tls") pod "dns-default-847hz" (UID: "4e09f673-9441-4afe-a258-85b6b5203de8") : secret "dns-default-metrics-tls" not found Jul 07 17:54:55.746984 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:55.746895 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wqfm4_997df85c-7632-47b5-a8a6-1c37f3346251/console-operator/0.log" Jul 07 17:54:55.746984 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:55.746951 2583 generic.go:356] "Generic (PLEG): container finished" podID="997df85c-7632-47b5-a8a6-1c37f3346251" containerID="d9c9f36bbd320128324a8e9a54143030468b263a1c48a89bc2d3faa5e92e0b93" exitCode=255 Jul 07 17:54:55.747449 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:55.747025 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" event={"ID":"997df85c-7632-47b5-a8a6-1c37f3346251","Type":"ContainerDied","Data":"d9c9f36bbd320128324a8e9a54143030468b263a1c48a89bc2d3faa5e92e0b93"} Jul 07 17:54:55.747449 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:55.747252 2583 scope.go:117] "RemoveContainer" containerID="d9c9f36bbd320128324a8e9a54143030468b263a1c48a89bc2d3faa5e92e0b93" Jul 07 17:54:55.748675 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:55.748644 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bjv6p" event={"ID":"2ed38cd7-9e63-4b1b-97cb-86748bb60a04","Type":"ContainerStarted","Data":"33ceb2eccac825f5a8096456f9c15020273f589f72565d3ac078383406a3f29e"} Jul 07 17:54:55.752278 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:55.752254 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-49qq2" event={"ID":"797a1f8a-8d68-4c01-a38b-67d405653bd0","Type":"ContainerStarted","Data":"400b066df4a990c55f013e2cbfc596d9bbea1123f9be46e55cb60efb5a1f614d"} Jul 07 17:54:55.803859 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:55.803811 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-49qq2" podStartSLOduration=10.252987781 podStartE2EDuration="43.803794126s" totalImagesPullingTime="33.550806345s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:54:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:54:55.801041828 +0000 UTC m=+44.883645531" watchObservedRunningTime="2026-07-07 17:54:55.803794126 +0000 UTC m=+44.886397831" Jul 07 17:54:55.826601 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:55.826544 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bjv6p" podStartSLOduration=3.188542399 podStartE2EDuration="5.826528174s" totalImagesPullingTime="2.637985775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:54:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:54:55.825727246 +0000 UTC m=+44.908330950" watchObservedRunningTime="2026-07-07 17:54:55.826528174 +0000 UTC m=+44.909131878" Jul 07 17:54:56.040726 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:56.040645 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:56.146148 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:56.146113 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:56.402921 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:56.402877 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:56.598637 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:56.598552 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:56.757695 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:56.757666 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wqfm4_997df85c-7632-47b5-a8a6-1c37f3346251/console-operator/1.log" Jul 07 17:54:56.758111 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:56.758090 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wqfm4_997df85c-7632-47b5-a8a6-1c37f3346251/console-operator/0.log" Jul 07 17:54:56.758160 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:56.758123 2583 generic.go:356] "Generic (PLEG): container finished" podID="997df85c-7632-47b5-a8a6-1c37f3346251" containerID="cb507ccae6942af4f5756b3e8c5b16cce1868d703d811f3505cf65553eb0951d" exitCode=255 Jul 07 17:54:56.758246 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:56.758221 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" event={"ID":"997df85c-7632-47b5-a8a6-1c37f3346251","Type":"ContainerDied","Data":"cb507ccae6942af4f5756b3e8c5b16cce1868d703d811f3505cf65553eb0951d"} Jul 07 17:54:56.758374 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:56.758311 2583 scope.go:117] "RemoveContainer" containerID="d9c9f36bbd320128324a8e9a54143030468b263a1c48a89bc2d3faa5e92e0b93" Jul 07 17:54:56.758583 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:56.758556 2583 scope.go:117] "RemoveContainer" containerID="cb507ccae6942af4f5756b3e8c5b16cce1868d703d811f3505cf65553eb0951d" Jul 07 17:54:56.758755 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:56.758719 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-wqfm4_openshift-console-operator(997df85c-7632-47b5-a8a6-1c37f3346251)\"" pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" podUID="997df85c-7632-47b5-a8a6-1c37f3346251" Jul 07 17:54:56.825041 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:56.825009 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:57.614420 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:57.614377 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:57.692636 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:57.692600 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:57.762303 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:57.762278 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wqfm4_997df85c-7632-47b5-a8a6-1c37f3346251/console-operator/1.log" Jul 07 17:54:57.762661 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:57.762613 2583 scope.go:117] "RemoveContainer" containerID="cb507ccae6942af4f5756b3e8c5b16cce1868d703d811f3505cf65553eb0951d" Jul 07 17:54:57.762799 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:57.762779 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-wqfm4_openshift-console-operator(997df85c-7632-47b5-a8a6-1c37f3346251)\"" pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" podUID="997df85c-7632-47b5-a8a6-1c37f3346251" Jul 07 17:54:58.582954 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:58.582885 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d9f9071-7f2b-4944-88fc-130da6a92ba5-service-ca-bundle\") pod \"router-default-6d876b7659-t4r6h\" (UID: \"4d9f9071-7f2b-4944-88fc-130da6a92ba5\") " pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:54:58.583138 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:58.582990 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d9f9071-7f2b-4944-88fc-130da6a92ba5-metrics-certs\") pod \"router-default-6d876b7659-t4r6h\" (UID: \"4d9f9071-7f2b-4944-88fc-130da6a92ba5\") " pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:54:58.583138 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:58.583089 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d9f9071-7f2b-4944-88fc-130da6a92ba5-service-ca-bundle podName:4d9f9071-7f2b-4944-88fc-130da6a92ba5 nodeName:}" failed. No retries permitted until 2026-07-07 17:55:06.583069754 +0000 UTC m=+55.665673436 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4d9f9071-7f2b-4944-88fc-130da6a92ba5-service-ca-bundle") pod "router-default-6d876b7659-t4r6h" (UID: "4d9f9071-7f2b-4944-88fc-130da6a92ba5") : configmap references non-existent config key: service-ca.crt Jul 07 17:54:58.583138 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:58.583104 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 17:54:58.583138 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:58.583138 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d9f9071-7f2b-4944-88fc-130da6a92ba5-metrics-certs podName:4d9f9071-7f2b-4944-88fc-130da6a92ba5 nodeName:}" failed. No retries permitted until 2026-07-07 17:55:06.583127319 +0000 UTC m=+55.665731015 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d9f9071-7f2b-4944-88fc-130da6a92ba5-metrics-certs") pod "router-default-6d876b7659-t4r6h" (UID: "4d9f9071-7f2b-4944-88fc-130da6a92ba5") : secret "router-metrics-certs-default" not found Jul 07 17:54:58.683618 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:58.683582 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-registry-tls\") pod \"image-registry-644cfc977-5rqx8\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:54:58.683756 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:58.683733 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 17:54:58.683809 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:58.683759 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-644cfc977-5rqx8: secret "image-registry-tls" not found Jul 07 17:54:58.683841 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:54:58.683813 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-registry-tls podName:13f33577-e89c-4c0f-9264-9a9971e8a17c nodeName:}" failed. No retries permitted until 2026-07-07 17:55:06.683797586 +0000 UTC m=+55.766401268 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-registry-tls") pod "image-registry-644cfc977-5rqx8" (UID: "13f33577-e89c-4c0f-9264-9a9971e8a17c") : secret "image-registry-tls" not found Jul 07 17:54:58.963976 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:58.963871 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:54:59.065878 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:54:59.065845 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:00.446881 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:00.446847 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-89b5k"] Jul 07 17:55:00.464312 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:00.464273 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-89b5k"] Jul 07 17:55:00.464467 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:00.464419 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-89b5k" Jul 07 17:55:00.467597 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:00.467573 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 07 17:55:00.467820 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:00.467800 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5jtj4\"" Jul 07 17:55:00.468539 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:00.468518 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 07 17:55:00.468539 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:00.468530 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 07 17:55:00.468685 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:00.468607 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 07 17:55:00.496429 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:00.496392 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:00.498337 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:00.498314 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/44101f7b-1922-44b1-9042-8a5bdc474c87-data-volume\") pod \"insights-runtime-extractor-89b5k\" (UID: \"44101f7b-1922-44b1-9042-8a5bdc474c87\") " pod="openshift-insights/insights-runtime-extractor-89b5k" Jul 07 17:55:00.498448 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:00.498403 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44101f7b-1922-44b1-9042-8a5bdc474c87-crio-socket\") pod \"insights-runtime-extractor-89b5k\" (UID: \"44101f7b-1922-44b1-9042-8a5bdc474c87\") " pod="openshift-insights/insights-runtime-extractor-89b5k" Jul 07 17:55:00.498493 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:00.498469 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44101f7b-1922-44b1-9042-8a5bdc474c87-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89b5k\" (UID: \"44101f7b-1922-44b1-9042-8a5bdc474c87\") " pod="openshift-insights/insights-runtime-extractor-89b5k" Jul 07 17:55:00.498536 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:00.498520 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44101f7b-1922-44b1-9042-8a5bdc474c87-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89b5k\" (UID: \"44101f7b-1922-44b1-9042-8a5bdc474c87\") " pod="openshift-insights/insights-runtime-extractor-89b5k" Jul 07 17:55:00.498615 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:00.498600 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z57dv\" (UniqueName: \"kubernetes.io/projected/44101f7b-1922-44b1-9042-8a5bdc474c87-kube-api-access-z57dv\") pod \"insights-runtime-extractor-89b5k\" (UID: \"44101f7b-1922-44b1-9042-8a5bdc474c87\") " pod="openshift-insights/insights-runtime-extractor-89b5k" Jul 07 17:55:00.579401 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:00.579369 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:00.599380 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:00.599342 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z57dv\" (UniqueName: \"kubernetes.io/projected/44101f7b-1922-44b1-9042-8a5bdc474c87-kube-api-access-z57dv\") pod \"insights-runtime-extractor-89b5k\" (UID: \"44101f7b-1922-44b1-9042-8a5bdc474c87\") " pod="openshift-insights/insights-runtime-extractor-89b5k" Jul 07 17:55:00.599380 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:00.599379 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/44101f7b-1922-44b1-9042-8a5bdc474c87-data-volume\") pod \"insights-runtime-extractor-89b5k\" (UID: \"44101f7b-1922-44b1-9042-8a5bdc474c87\") " pod="openshift-insights/insights-runtime-extractor-89b5k" Jul 07 17:55:00.599640 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:00.599450 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44101f7b-1922-44b1-9042-8a5bdc474c87-crio-socket\") pod \"insights-runtime-extractor-89b5k\" (UID: \"44101f7b-1922-44b1-9042-8a5bdc474c87\") " pod="openshift-insights/insights-runtime-extractor-89b5k" Jul 07 17:55:00.599640 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:00.599484 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44101f7b-1922-44b1-9042-8a5bdc474c87-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89b5k\" (UID: \"44101f7b-1922-44b1-9042-8a5bdc474c87\") " pod="openshift-insights/insights-runtime-extractor-89b5k" Jul 07 17:55:00.599640 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:00.599535 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44101f7b-1922-44b1-9042-8a5bdc474c87-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89b5k\" (UID: \"44101f7b-1922-44b1-9042-8a5bdc474c87\") " pod="openshift-insights/insights-runtime-extractor-89b5k" Jul 07 17:55:00.599786 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:00.599660 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 17:55:00.599786 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:00.599662 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 17:55:00.599786 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:00.599725 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/44101f7b-1922-44b1-9042-8a5bdc474c87-crio-socket\") pod \"insights-runtime-extractor-89b5k\" (UID: \"44101f7b-1922-44b1-9042-8a5bdc474c87\") " pod="openshift-insights/insights-runtime-extractor-89b5k" Jul 07 17:55:00.599786 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:00.599751 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44101f7b-1922-44b1-9042-8a5bdc474c87-kube-rbac-proxy-cm podName:44101f7b-1922-44b1-9042-8a5bdc474c87 nodeName:}" failed. No retries permitted until 2026-07-07 17:55:01.09972788 +0000 UTC m=+50.182331580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44101f7b-1922-44b1-9042-8a5bdc474c87-kube-rbac-proxy-cm") pod "insights-runtime-extractor-89b5k" (UID: "44101f7b-1922-44b1-9042-8a5bdc474c87") : configmap "kube-rbac-proxy" not found Jul 07 17:55:00.599786 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:00.599778 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44101f7b-1922-44b1-9042-8a5bdc474c87-insights-runtime-extractor-tls podName:44101f7b-1922-44b1-9042-8a5bdc474c87 nodeName:}" failed. No retries permitted until 2026-07-07 17:55:01.099760941 +0000 UTC m=+50.182364640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44101f7b-1922-44b1-9042-8a5bdc474c87-insights-runtime-extractor-tls") pod "insights-runtime-extractor-89b5k" (UID: "44101f7b-1922-44b1-9042-8a5bdc474c87") : secret "insights-runtime-extractor-tls" not found Jul 07 17:55:00.600055 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:00.599829 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/44101f7b-1922-44b1-9042-8a5bdc474c87-data-volume\") pod \"insights-runtime-extractor-89b5k\" (UID: \"44101f7b-1922-44b1-9042-8a5bdc474c87\") " pod="openshift-insights/insights-runtime-extractor-89b5k" Jul 07 17:55:00.615922 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:00.615877 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z57dv\" (UniqueName: \"kubernetes.io/projected/44101f7b-1922-44b1-9042-8a5bdc474c87-kube-api-access-z57dv\") pod \"insights-runtime-extractor-89b5k\" (UID: \"44101f7b-1922-44b1-9042-8a5bdc474c87\") " pod="openshift-insights/insights-runtime-extractor-89b5k" Jul 07 17:55:00.769616 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:00.769533 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:00.840008 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:00.839970 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:01.103984 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:01.103953 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44101f7b-1922-44b1-9042-8a5bdc474c87-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89b5k\" (UID: \"44101f7b-1922-44b1-9042-8a5bdc474c87\") " pod="openshift-insights/insights-runtime-extractor-89b5k" Jul 07 17:55:01.104143 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:01.104006 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44101f7b-1922-44b1-9042-8a5bdc474c87-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89b5k\" (UID: \"44101f7b-1922-44b1-9042-8a5bdc474c87\") " pod="openshift-insights/insights-runtime-extractor-89b5k" Jul 07 17:55:01.104143 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:01.104064 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 17:55:01.104143 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:01.104139 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44101f7b-1922-44b1-9042-8a5bdc474c87-kube-rbac-proxy-cm podName:44101f7b-1922-44b1-9042-8a5bdc474c87 nodeName:}" failed. No retries permitted until 2026-07-07 17:55:02.104118655 +0000 UTC m=+51.186722338 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44101f7b-1922-44b1-9042-8a5bdc474c87-kube-rbac-proxy-cm") pod "insights-runtime-extractor-89b5k" (UID: "44101f7b-1922-44b1-9042-8a5bdc474c87") : configmap "kube-rbac-proxy" not found Jul 07 17:55:01.104257 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:01.104165 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 17:55:01.104257 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:01.104224 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44101f7b-1922-44b1-9042-8a5bdc474c87-insights-runtime-extractor-tls podName:44101f7b-1922-44b1-9042-8a5bdc474c87 nodeName:}" failed. No retries permitted until 2026-07-07 17:55:02.104210524 +0000 UTC m=+51.186814228 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44101f7b-1922-44b1-9042-8a5bdc474c87-insights-runtime-extractor-tls") pod "insights-runtime-extractor-89b5k" (UID: "44101f7b-1922-44b1-9042-8a5bdc474c87") : secret "insights-runtime-extractor-tls" not found Jul 07 17:55:01.203019 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:01.202980 2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" Jul 07 17:55:01.203204 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:01.203050 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" Jul 07 17:55:01.203455 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:01.203438 2583 scope.go:117] "RemoveContainer" containerID="cb507ccae6942af4f5756b3e8c5b16cce1868d703d811f3505cf65553eb0951d" Jul 07 17:55:01.203632 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:01.203608 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-wqfm4_openshift-console-operator(997df85c-7632-47b5-a8a6-1c37f3346251)\"" pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" podUID="997df85c-7632-47b5-a8a6-1c37f3346251" Jul 07 17:55:01.532004 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:01.531887 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:01.637329 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:01.637290 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:01.761040 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:01.761009 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:01.861483 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:01.861437 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:02.111781 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:02.111700 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44101f7b-1922-44b1-9042-8a5bdc474c87-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89b5k\" (UID: \"44101f7b-1922-44b1-9042-8a5bdc474c87\") " pod="openshift-insights/insights-runtime-extractor-89b5k" Jul 07 17:55:02.111781 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:02.111750 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44101f7b-1922-44b1-9042-8a5bdc474c87-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89b5k\" (UID: \"44101f7b-1922-44b1-9042-8a5bdc474c87\") " pod="openshift-insights/insights-runtime-extractor-89b5k" Jul 07 17:55:02.111981 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:02.111833 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 17:55:02.111981 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:02.111921 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44101f7b-1922-44b1-9042-8a5bdc474c87-kube-rbac-proxy-cm podName:44101f7b-1922-44b1-9042-8a5bdc474c87 nodeName:}" failed. No retries permitted until 2026-07-07 17:55:04.111885683 +0000 UTC m=+53.194489366 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44101f7b-1922-44b1-9042-8a5bdc474c87-kube-rbac-proxy-cm") pod "insights-runtime-extractor-89b5k" (UID: "44101f7b-1922-44b1-9042-8a5bdc474c87") : configmap "kube-rbac-proxy" not found Jul 07 17:55:02.111981 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:02.111926 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 17:55:02.111981 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:02.111978 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44101f7b-1922-44b1-9042-8a5bdc474c87-insights-runtime-extractor-tls podName:44101f7b-1922-44b1-9042-8a5bdc474c87 nodeName:}" failed. No retries permitted until 2026-07-07 17:55:04.111963371 +0000 UTC m=+53.194567068 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44101f7b-1922-44b1-9042-8a5bdc474c87-insights-runtime-extractor-tls") pod "insights-runtime-extractor-89b5k" (UID: "44101f7b-1922-44b1-9042-8a5bdc474c87") : secret "insights-runtime-extractor-tls" not found Jul 07 17:55:03.121804 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:03.121763 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e09f673-9441-4afe-a258-85b6b5203de8-metrics-tls\") pod \"dns-default-847hz\" (UID: \"4e09f673-9441-4afe-a258-85b6b5203de8\") " pod="openshift-dns/dns-default-847hz" Jul 07 17:55:03.122241 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:03.121942 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 17:55:03.122241 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:03.122009 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e09f673-9441-4afe-a258-85b6b5203de8-metrics-tls podName:4e09f673-9441-4afe-a258-85b6b5203de8 nodeName:}" failed. No retries permitted until 2026-07-07 17:55:19.121992135 +0000 UTC m=+68.204595821 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e09f673-9441-4afe-a258-85b6b5203de8-metrics-tls") pod "dns-default-847hz" (UID: "4e09f673-9441-4afe-a258-85b6b5203de8") : secret "dns-default-metrics-tls" not found Jul 07 17:55:03.132592 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:03.132557 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:03.226586 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:03.226557 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p4mnt_e5c3b9f0-ea6c-4262-b4e7-88945edc9f6f/dns-node-resolver/0.log" Jul 07 17:55:03.232493 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:03.232458 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:04.030306 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:04.030277 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dm6hm_d5ea083a-f7fb-4e82-a02a-b930204910b6/node-ca/0.log" Jul 07 17:55:04.128690 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:04.128650 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44101f7b-1922-44b1-9042-8a5bdc474c87-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89b5k\" (UID: \"44101f7b-1922-44b1-9042-8a5bdc474c87\") " pod="openshift-insights/insights-runtime-extractor-89b5k" Jul 07 17:55:04.129086 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:04.128705 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44101f7b-1922-44b1-9042-8a5bdc474c87-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89b5k\" (UID: \"44101f7b-1922-44b1-9042-8a5bdc474c87\") " pod="openshift-insights/insights-runtime-extractor-89b5k" Jul 07 17:55:04.129086 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:04.128778 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 17:55:04.129086 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:04.128799 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 17:55:04.129086 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:04.128837 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44101f7b-1922-44b1-9042-8a5bdc474c87-kube-rbac-proxy-cm podName:44101f7b-1922-44b1-9042-8a5bdc474c87 nodeName:}" failed. No retries permitted until 2026-07-07 17:55:08.12882249 +0000 UTC m=+57.211426171 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44101f7b-1922-44b1-9042-8a5bdc474c87-kube-rbac-proxy-cm") pod "insights-runtime-extractor-89b5k" (UID: "44101f7b-1922-44b1-9042-8a5bdc474c87") : configmap "kube-rbac-proxy" not found Jul 07 17:55:04.129086 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:04.128856 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44101f7b-1922-44b1-9042-8a5bdc474c87-insights-runtime-extractor-tls podName:44101f7b-1922-44b1-9042-8a5bdc474c87 nodeName:}" failed. No retries permitted until 2026-07-07 17:55:08.128843864 +0000 UTC m=+57.211447545 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44101f7b-1922-44b1-9042-8a5bdc474c87-insights-runtime-extractor-tls") pod "insights-runtime-extractor-89b5k" (UID: "44101f7b-1922-44b1-9042-8a5bdc474c87") : secret "insights-runtime-extractor-tls" not found Jul 07 17:55:04.238164 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:04.238130 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:05.025969 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:05.025934 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dr6hd_686b9d0e-9891-4e46-b78b-c5030ceffa06/serve-healthcheck-canary/0.log" Jul 07 17:55:05.829572 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:05.829543 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-bjv6p_2ed38cd7-9e63-4b1b-97cb-86748bb60a04/kube-storage-version-migrator-operator/0.log" Jul 07 17:55:06.649187 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:06.649137 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d9f9071-7f2b-4944-88fc-130da6a92ba5-metrics-certs\") pod \"router-default-6d876b7659-t4r6h\" (UID: \"4d9f9071-7f2b-4944-88fc-130da6a92ba5\") " pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:55:06.649449 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:06.649261 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d9f9071-7f2b-4944-88fc-130da6a92ba5-service-ca-bundle\") pod \"router-default-6d876b7659-t4r6h\" (UID: \"4d9f9071-7f2b-4944-88fc-130da6a92ba5\") " pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:55:06.649449 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:06.649317 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 17:55:06.649449 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:06.649371 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4d9f9071-7f2b-4944-88fc-130da6a92ba5-service-ca-bundle podName:4d9f9071-7f2b-4944-88fc-130da6a92ba5 nodeName:}" failed. No retries permitted until 2026-07-07 17:55:22.64935501 +0000 UTC m=+71.731958691 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4d9f9071-7f2b-4944-88fc-130da6a92ba5-service-ca-bundle") pod "router-default-6d876b7659-t4r6h" (UID: "4d9f9071-7f2b-4944-88fc-130da6a92ba5") : configmap references non-existent config key: service-ca.crt Jul 07 17:55:06.649449 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:06.649385 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d9f9071-7f2b-4944-88fc-130da6a92ba5-metrics-certs podName:4d9f9071-7f2b-4944-88fc-130da6a92ba5 nodeName:}" failed. No retries permitted until 2026-07-07 17:55:22.649379337 +0000 UTC m=+71.731983019 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d9f9071-7f2b-4944-88fc-130da6a92ba5-metrics-certs") pod "router-default-6d876b7659-t4r6h" (UID: "4d9f9071-7f2b-4944-88fc-130da6a92ba5") : secret "router-metrics-certs-default" not found Jul 07 17:55:06.750143 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:06.750039 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-registry-tls\") pod \"image-registry-644cfc977-5rqx8\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:55:06.750143 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:06.750145 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 17:55:06.750330 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:06.750163 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-644cfc977-5rqx8: secret "image-registry-tls" not found Jul 07 17:55:06.750330 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:06.750222 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-registry-tls podName:13f33577-e89c-4c0f-9264-9a9971e8a17c nodeName:}" failed. No retries permitted until 2026-07-07 17:55:22.750208775 +0000 UTC m=+71.832812456 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-registry-tls") pod "image-registry-644cfc977-5rqx8" (UID: "13f33577-e89c-4c0f-9264-9a9971e8a17c") : secret "image-registry-tls" not found Jul 07 17:55:07.081436 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:07.081394 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:08.030469 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:08.030440 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:08.157993 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:08.157952 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:08.163001 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:08.162979 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44101f7b-1922-44b1-9042-8a5bdc474c87-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89b5k\" (UID: \"44101f7b-1922-44b1-9042-8a5bdc474c87\") " pod="openshift-insights/insights-runtime-extractor-89b5k" Jul 07 17:55:08.163061 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:08.163025 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44101f7b-1922-44b1-9042-8a5bdc474c87-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89b5k\" (UID: \"44101f7b-1922-44b1-9042-8a5bdc474c87\") " pod="openshift-insights/insights-runtime-extractor-89b5k" Jul 07 17:55:08.163100 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:08.163092 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 17:55:08.163145 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:08.163134 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 17:55:08.163179 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:08.163149 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44101f7b-1922-44b1-9042-8a5bdc474c87-kube-rbac-proxy-cm podName:44101f7b-1922-44b1-9042-8a5bdc474c87 nodeName:}" failed. No retries permitted until 2026-07-07 17:55:16.163133925 +0000 UTC m=+65.245737607 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/44101f7b-1922-44b1-9042-8a5bdc474c87-kube-rbac-proxy-cm") pod "insights-runtime-extractor-89b5k" (UID: "44101f7b-1922-44b1-9042-8a5bdc474c87") : configmap "kube-rbac-proxy" not found Jul 07 17:55:08.163218 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:08.163187 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44101f7b-1922-44b1-9042-8a5bdc474c87-insights-runtime-extractor-tls podName:44101f7b-1922-44b1-9042-8a5bdc474c87 nodeName:}" failed. No retries permitted until 2026-07-07 17:55:16.163175971 +0000 UTC m=+65.245779653 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44101f7b-1922-44b1-9042-8a5bdc474c87-insights-runtime-extractor-tls") pod "insights-runtime-extractor-89b5k" (UID: "44101f7b-1922-44b1-9042-8a5bdc474c87") : secret "insights-runtime-extractor-tls" not found Jul 07 17:55:08.270150 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:08.270077 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:08.384847 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:08.384813 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:08.538132 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:08.538090 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:08.638283 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:08.638189 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:08.751609 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:08.751559 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:09.680877 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:09.680844 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:09.753458 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:09.753421 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:09.915819 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:09.915781 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:10.119403 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:10.119368 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:10.550959 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:10.550920 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:11.012035 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:11.011942 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:11.146224 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:11.146193 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:11.535042 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:11.535001 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:11.739369 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:11.739330 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:11.821073 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:11.821041 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:12.101306 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:12.101221 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:12.192218 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:12.192187 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:12.716234 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:12.716199 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:12.819572 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:12.819540 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:12.919323 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:12.919288 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:13.120203 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:13.120164 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:13.521545 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:13.521453 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:13.703070 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:13.703046 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zjzpw" Jul 07 17:55:13.732492 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:13.732422 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:14.360648 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:14.360616 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:14.460438 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:14.460405 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:14.561598 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:14.561565 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:14.919087 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:14.919048 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:15.340415 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:15.340374 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:16.185588 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:16.185555 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:16.238955 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:16.238887 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44101f7b-1922-44b1-9042-8a5bdc474c87-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89b5k\" (UID: \"44101f7b-1922-44b1-9042-8a5bdc474c87\") " pod="openshift-insights/insights-runtime-extractor-89b5k" Jul 07 17:55:16.239133 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:16.239050 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 17:55:16.239133 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:16.239057 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44101f7b-1922-44b1-9042-8a5bdc474c87-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89b5k\" (UID: \"44101f7b-1922-44b1-9042-8a5bdc474c87\") " pod="openshift-insights/insights-runtime-extractor-89b5k" Jul 07 17:55:16.239133 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:16.239108 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44101f7b-1922-44b1-9042-8a5bdc474c87-insights-runtime-extractor-tls podName:44101f7b-1922-44b1-9042-8a5bdc474c87 nodeName:}" failed. No retries permitted until 2026-07-07 17:55:32.239092821 +0000 UTC m=+81.321696503 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/44101f7b-1922-44b1-9042-8a5bdc474c87-insights-runtime-extractor-tls") pod "insights-runtime-extractor-89b5k" (UID: "44101f7b-1922-44b1-9042-8a5bdc474c87") : secret "insights-runtime-extractor-tls" not found Jul 07 17:55:16.239538 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:16.239515 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/44101f7b-1922-44b1-9042-8a5bdc474c87-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89b5k\" (UID: \"44101f7b-1922-44b1-9042-8a5bdc474c87\") " pod="openshift-insights/insights-runtime-extractor-89b5k" Jul 07 17:55:16.344770 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:16.344737 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:16.423881 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:16.423849 2583 scope.go:117] "RemoveContainer" containerID="cb507ccae6942af4f5756b3e8c5b16cce1868d703d811f3505cf65553eb0951d" Jul 07 17:55:16.735915 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:16.735814 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:16.816106 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:16.816080 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wqfm4_997df85c-7632-47b5-a8a6-1c37f3346251/console-operator/2.log" Jul 07 17:55:16.816470 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:16.816452 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wqfm4_997df85c-7632-47b5-a8a6-1c37f3346251/console-operator/1.log" Jul 07 17:55:16.816518 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:16.816488 2583 generic.go:356] "Generic (PLEG): container finished" podID="997df85c-7632-47b5-a8a6-1c37f3346251" containerID="e6b1245a376909f6b71c28cdd2f927c690e644ffe4e5422b0b5b82ad4136de58" exitCode=255 Jul 07 17:55:16.816559 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:16.816546 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" event={"ID":"997df85c-7632-47b5-a8a6-1c37f3346251","Type":"ContainerDied","Data":"e6b1245a376909f6b71c28cdd2f927c690e644ffe4e5422b0b5b82ad4136de58"} Jul 07 17:55:16.816602 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:16.816593 2583 scope.go:117] "RemoveContainer" containerID="cb507ccae6942af4f5756b3e8c5b16cce1868d703d811f3505cf65553eb0951d" Jul 07 17:55:16.816943 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:16.816921 2583 scope.go:117] "RemoveContainer" containerID="e6b1245a376909f6b71c28cdd2f927c690e644ffe4e5422b0b5b82ad4136de58" Jul 07 17:55:16.817092 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:16.817067 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-wqfm4_openshift-console-operator(997df85c-7632-47b5-a8a6-1c37f3346251)\"" pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" podUID="997df85c-7632-47b5-a8a6-1c37f3346251" Jul 07 17:55:16.929217 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:16.929148 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:17.148672 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:17.148631 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a1d71c1-6f70-441c-87bb-1f997df1fd5d-original-pull-secret\") pod \"global-pull-secret-syncer-xqq9h\" (UID: \"3a1d71c1-6f70-441c-87bb-1f997df1fd5d\") " pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:55:17.148841 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:17.148742 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2fbb\" (UniqueName: \"kubernetes.io/projected/8a7f4631-8582-4636-832f-e026bb298b27-kube-api-access-g2fbb\") pod \"network-check-target-bmdgm\" (UID: \"8a7f4631-8582-4636-832f-e026bb298b27\") " pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:55:17.153247 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:17.153222 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 17:55:17.153378 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:17.153230 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 17:55:17.162026 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:17.162004 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a1d71c1-6f70-441c-87bb-1f997df1fd5d-original-pull-secret\") pod \"global-pull-secret-syncer-xqq9h\" (UID: \"3a1d71c1-6f70-441c-87bb-1f997df1fd5d\") " pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:55:17.165550 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:17.165528 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 17:55:17.172720 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:17.172693 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2fbb\" (UniqueName: \"kubernetes.io/projected/8a7f4631-8582-4636-832f-e026bb298b27-kube-api-access-g2fbb\") pod \"network-check-target-bmdgm\" (UID: \"8a7f4631-8582-4636-832f-e026bb298b27\") " pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:55:17.249096 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:17.249066 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n28xq\"" Jul 07 17:55:17.249494 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:17.249189 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e10d7dd-5837-417c-bbe3-2cbf4ca1c144-metrics-certs\") pod \"network-metrics-daemon-lwjzj\" (UID: \"6e10d7dd-5837-417c-bbe3-2cbf4ca1c144\") " pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:55:17.251143 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:17.251127 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xqq9h" Jul 07 17:55:17.252500 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:17.252485 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 17:55:17.255799 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:17.255781 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:55:17.262205 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:17.262162 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e10d7dd-5837-417c-bbe3-2cbf4ca1c144-metrics-certs\") pod \"network-metrics-daemon-lwjzj\" (UID: \"6e10d7dd-5837-417c-bbe3-2cbf4ca1c144\") " pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:55:17.330211 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:17.330172 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:17.416512 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:17.416341 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xqq9h"] Jul 07 17:55:17.418884 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:55:17.418856 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1d71c1_6f70_441c_87bb_1f997df1fd5d.slice/crio-fcac4f3d4228483799cd6e5adc350b863636b193013dcd2bcee6ff0f2db8b8ab WatchSource:0}: Error finding container fcac4f3d4228483799cd6e5adc350b863636b193013dcd2bcee6ff0f2db8b8ab: Status 404 returned error can't find the container with id fcac4f3d4228483799cd6e5adc350b863636b193013dcd2bcee6ff0f2db8b8ab Jul 07 17:55:17.434074 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:17.431097 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bmdgm"] Jul 07 17:55:17.540539 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:17.540505 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g4wsg\"" Jul 07 17:55:17.547072 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:17.547043 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwjzj" Jul 07 17:55:17.676995 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:17.676919 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lwjzj"] Jul 07 17:55:17.679846 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:55:17.679816 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e10d7dd_5837_417c_bbe3_2cbf4ca1c144.slice/crio-53a59b1e4e269ef3e76e2e49fbf7482dcf7a72db66a5ee066d012e01a7db37d9 WatchSource:0}: Error finding container 53a59b1e4e269ef3e76e2e49fbf7482dcf7a72db66a5ee066d012e01a7db37d9: Status 404 returned error can't find the container with id 53a59b1e4e269ef3e76e2e49fbf7482dcf7a72db66a5ee066d012e01a7db37d9 Jul 07 17:55:17.729489 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:17.729460 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:17.820669 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:17.820642 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wqfm4_997df85c-7632-47b5-a8a6-1c37f3346251/console-operator/2.log" Jul 07 17:55:17.821822 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:17.821790 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xqq9h" event={"ID":"3a1d71c1-6f70-441c-87bb-1f997df1fd5d","Type":"ContainerStarted","Data":"fcac4f3d4228483799cd6e5adc350b863636b193013dcd2bcee6ff0f2db8b8ab"} Jul 07 17:55:17.823034 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:17.823011 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lwjzj" event={"ID":"6e10d7dd-5837-417c-bbe3-2cbf4ca1c144","Type":"ContainerStarted","Data":"53a59b1e4e269ef3e76e2e49fbf7482dcf7a72db66a5ee066d012e01a7db37d9"} Jul 07 17:55:17.824107 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:17.824086 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bmdgm" event={"ID":"8a7f4631-8582-4636-832f-e026bb298b27","Type":"ContainerStarted","Data":"532efdd76fdbbf69d1ee6d3be104f1526dac9812e1a35b5ff9c91298cded72b5"} Jul 07 17:55:18.371884 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:18.371842 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:18.522411 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:18.522298 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:18.743765 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:18.743672 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:19.168099 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:19.166715 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e09f673-9441-4afe-a258-85b6b5203de8-metrics-tls\") pod \"dns-default-847hz\" (UID: \"4e09f673-9441-4afe-a258-85b6b5203de8\") " pod="openshift-dns/dns-default-847hz" Jul 07 17:55:19.169573 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:19.169544 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e09f673-9441-4afe-a258-85b6b5203de8-metrics-tls\") pod \"dns-default-847hz\" (UID: \"4e09f673-9441-4afe-a258-85b6b5203de8\") " pod="openshift-dns/dns-default-847hz" Jul 07 17:55:19.406773 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:19.406690 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dtpjm\"" Jul 07 17:55:19.414090 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:19.414061 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-847hz" Jul 07 17:55:19.574932 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:19.574868 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-847hz"] Jul 07 17:55:19.578706 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:55:19.578672 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e09f673_9441_4afe_a258_85b6b5203de8.slice/crio-2874c3dbde4368a55ac7387b40c8d81460a72d9bda1b7860af25334957a3a56c WatchSource:0}: Error finding container 2874c3dbde4368a55ac7387b40c8d81460a72d9bda1b7860af25334957a3a56c: Status 404 returned error can't find the container with id 2874c3dbde4368a55ac7387b40c8d81460a72d9bda1b7860af25334957a3a56c Jul 07 17:55:19.834201 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:19.834161 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lwjzj" event={"ID":"6e10d7dd-5837-417c-bbe3-2cbf4ca1c144","Type":"ContainerStarted","Data":"fbc45c6eab0a7ea0026a23a8a6de1baabe804603bfed5944b0b76dede5c516f0"} Jul 07 17:55:19.834398 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:19.834232 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lwjzj" event={"ID":"6e10d7dd-5837-417c-bbe3-2cbf4ca1c144","Type":"ContainerStarted","Data":"226635861e9cec9d6d1e8484611fe46d19966ee5b5c57b8a62c270018c1db061"} Jul 07 17:55:19.835676 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:19.835649 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-847hz" event={"ID":"4e09f673-9441-4afe-a258-85b6b5203de8","Type":"ContainerStarted","Data":"2874c3dbde4368a55ac7387b40c8d81460a72d9bda1b7860af25334957a3a56c"} Jul 07 17:55:19.853320 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:19.853264 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lwjzj" podStartSLOduration=66.466706896 podStartE2EDuration="1m7.853245558s" totalImagesPullingTime="1.386538662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:55:19.852632333 +0000 UTC m=+68.935236038" watchObservedRunningTime="2026-07-07 17:55:19.853245558 +0000 UTC m=+68.935849262" Jul 07 17:55:21.203476 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:21.203435 2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" Jul 07 17:55:21.203952 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:21.203496 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" Jul 07 17:55:21.203952 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:21.203867 2583 scope.go:117] "RemoveContainer" containerID="e6b1245a376909f6b71c28cdd2f927c690e644ffe4e5422b0b5b82ad4136de58" Jul 07 17:55:21.204101 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:21.204079 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-wqfm4_openshift-console-operator(997df85c-7632-47b5-a8a6-1c37f3346251)\"" pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" podUID="997df85c-7632-47b5-a8a6-1c37f3346251" Jul 07 17:55:22.700130 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:22.700074 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d9f9071-7f2b-4944-88fc-130da6a92ba5-service-ca-bundle\") pod \"router-default-6d876b7659-t4r6h\" (UID: \"4d9f9071-7f2b-4944-88fc-130da6a92ba5\") " pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:55:22.700663 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:22.700145 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d9f9071-7f2b-4944-88fc-130da6a92ba5-metrics-certs\") pod \"router-default-6d876b7659-t4r6h\" (UID: \"4d9f9071-7f2b-4944-88fc-130da6a92ba5\") " pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:55:22.701181 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:22.701153 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d9f9071-7f2b-4944-88fc-130da6a92ba5-service-ca-bundle\") pod \"router-default-6d876b7659-t4r6h\" (UID: \"4d9f9071-7f2b-4944-88fc-130da6a92ba5\") " pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:55:22.702994 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:22.702972 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d9f9071-7f2b-4944-88fc-130da6a92ba5-metrics-certs\") pod \"router-default-6d876b7659-t4r6h\" (UID: \"4d9f9071-7f2b-4944-88fc-130da6a92ba5\") " pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:55:22.788343 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:22.788304 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-644cfc977-5rqx8"] Jul 07 17:55:22.788648 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:22.788621 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-644cfc977-5rqx8" podUID="13f33577-e89c-4c0f-9264-9a9971e8a17c" Jul 07 17:55:22.800977 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:22.800938 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-registry-tls\") pod \"image-registry-644cfc977-5rqx8\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:55:22.803932 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:22.803887 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-registry-tls\") pod \"image-registry-644cfc977-5rqx8\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:55:22.846621 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:22.846589 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:55:22.853324 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:22.853299 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:55:22.918115 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:22.918088 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-rfvlw\"" Jul 07 17:55:22.920175 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:22.920152 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:55:22.992356 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:22.992282 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-77c4ff87fd-tjwtz"] Jul 07 17:55:22.994246 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:22.994226 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77c4ff87fd-tjwtz" Jul 07 17:55:23.002209 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.002185 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87kzf\" (UniqueName: \"kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-kube-api-access-87kzf\") pod \"13f33577-e89c-4c0f-9264-9a9971e8a17c\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " Jul 07 17:55:23.002347 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.002225 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13f33577-e89c-4c0f-9264-9a9971e8a17c-ca-trust-extracted\") pod \"13f33577-e89c-4c0f-9264-9a9971e8a17c\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " Jul 07 17:55:23.002347 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.002250 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13f33577-e89c-4c0f-9264-9a9971e8a17c-trusted-ca\") pod \"13f33577-e89c-4c0f-9264-9a9971e8a17c\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " Jul 07 17:55:23.002462 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.002358 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-bound-sa-token\") pod \"13f33577-e89c-4c0f-9264-9a9971e8a17c\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " Jul 07 17:55:23.002462 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.002394 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-registry-tls\") pod \"13f33577-e89c-4c0f-9264-9a9971e8a17c\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " Jul 07 17:55:23.002462 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.002426 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13f33577-e89c-4c0f-9264-9a9971e8a17c-installation-pull-secrets\") pod \"13f33577-e89c-4c0f-9264-9a9971e8a17c\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " Jul 07 17:55:23.002462 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.002447 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13f33577-e89c-4c0f-9264-9a9971e8a17c-registry-certificates\") pod \"13f33577-e89c-4c0f-9264-9a9971e8a17c\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " Jul 07 17:55:23.002652 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.002505 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f33577-e89c-4c0f-9264-9a9971e8a17c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "13f33577-e89c-4c0f-9264-9a9971e8a17c" (UID: "13f33577-e89c-4c0f-9264-9a9971e8a17c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:55:23.002652 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.002523 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13f33577-e89c-4c0f-9264-9a9971e8a17c-image-registry-private-configuration\") pod \"13f33577-e89c-4c0f-9264-9a9971e8a17c\" (UID: \"13f33577-e89c-4c0f-9264-9a9971e8a17c\") " Jul 07 17:55:23.002652 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.002595 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f33577-e89c-4c0f-9264-9a9971e8a17c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "13f33577-e89c-4c0f-9264-9a9971e8a17c" (UID: "13f33577-e89c-4c0f-9264-9a9971e8a17c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:55:23.002802 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.002726 2583 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13f33577-e89c-4c0f-9264-9a9971e8a17c-ca-trust-extracted\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:55:23.002802 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.002743 2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13f33577-e89c-4c0f-9264-9a9971e8a17c-trusted-ca\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:55:23.003132 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.003108 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f33577-e89c-4c0f-9264-9a9971e8a17c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "13f33577-e89c-4c0f-9264-9a9971e8a17c" (UID: "13f33577-e89c-4c0f-9264-9a9971e8a17c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:55:23.004841 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.004816 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-kube-api-access-87kzf" (OuterVolumeSpecName: "kube-api-access-87kzf") pod "13f33577-e89c-4c0f-9264-9a9971e8a17c" (UID: "13f33577-e89c-4c0f-9264-9a9971e8a17c"). InnerVolumeSpecName "kube-api-access-87kzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:55:23.005132 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.005105 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "13f33577-e89c-4c0f-9264-9a9971e8a17c" (UID: "13f33577-e89c-4c0f-9264-9a9971e8a17c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:55:23.005209 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.005192 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13f33577-e89c-4c0f-9264-9a9971e8a17c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "13f33577-e89c-4c0f-9264-9a9971e8a17c" (UID: "13f33577-e89c-4c0f-9264-9a9971e8a17c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:55:23.005346 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.005322 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "13f33577-e89c-4c0f-9264-9a9971e8a17c" (UID: "13f33577-e89c-4c0f-9264-9a9971e8a17c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:55:23.005387 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.005366 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13f33577-e89c-4c0f-9264-9a9971e8a17c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "13f33577-e89c-4c0f-9264-9a9971e8a17c" (UID: "13f33577-e89c-4c0f-9264-9a9971e8a17c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:55:23.013479 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.013455 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77c4ff87fd-tjwtz"] Jul 07 17:55:23.093632 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.093581 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:23.104652 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.104118 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4cb6eb72-4869-400a-a56d-d712f4c8ff83-ca-trust-extracted\") pod \"image-registry-77c4ff87fd-tjwtz\" (UID: \"4cb6eb72-4869-400a-a56d-d712f4c8ff83\") " pod="openshift-image-registry/image-registry-77c4ff87fd-tjwtz" Jul 07 17:55:23.104652 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.104168 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4cb6eb72-4869-400a-a56d-d712f4c8ff83-trusted-ca\") pod \"image-registry-77c4ff87fd-tjwtz\" (UID: \"4cb6eb72-4869-400a-a56d-d712f4c8ff83\") " pod="openshift-image-registry/image-registry-77c4ff87fd-tjwtz" Jul 07 17:55:23.104652 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.104201 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4cb6eb72-4869-400a-a56d-d712f4c8ff83-image-registry-private-configuration\") pod \"image-registry-77c4ff87fd-tjwtz\" (UID: \"4cb6eb72-4869-400a-a56d-d712f4c8ff83\") " pod="openshift-image-registry/image-registry-77c4ff87fd-tjwtz" Jul 07 17:55:23.104652 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.104228 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cb6eb72-4869-400a-a56d-d712f4c8ff83-registry-tls\") pod \"image-registry-77c4ff87fd-tjwtz\" (UID: \"4cb6eb72-4869-400a-a56d-d712f4c8ff83\") " pod="openshift-image-registry/image-registry-77c4ff87fd-tjwtz" Jul 07 17:55:23.104652 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.104266 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4cb6eb72-4869-400a-a56d-d712f4c8ff83-installation-pull-secrets\") pod \"image-registry-77c4ff87fd-tjwtz\" (UID: \"4cb6eb72-4869-400a-a56d-d712f4c8ff83\") " pod="openshift-image-registry/image-registry-77c4ff87fd-tjwtz" Jul 07 17:55:23.104652 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.104337 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4cb6eb72-4869-400a-a56d-d712f4c8ff83-bound-sa-token\") pod \"image-registry-77c4ff87fd-tjwtz\" (UID: \"4cb6eb72-4869-400a-a56d-d712f4c8ff83\") " pod="openshift-image-registry/image-registry-77c4ff87fd-tjwtz" Jul 07 17:55:23.104652 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.104361 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4cb6eb72-4869-400a-a56d-d712f4c8ff83-registry-certificates\") pod \"image-registry-77c4ff87fd-tjwtz\" (UID: \"4cb6eb72-4869-400a-a56d-d712f4c8ff83\") " pod="openshift-image-registry/image-registry-77c4ff87fd-tjwtz" Jul 07 17:55:23.104652 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.104403 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djtp9\" (UniqueName: \"kubernetes.io/projected/4cb6eb72-4869-400a-a56d-d712f4c8ff83-kube-api-access-djtp9\") pod \"image-registry-77c4ff87fd-tjwtz\" (UID: \"4cb6eb72-4869-400a-a56d-d712f4c8ff83\") " pod="openshift-image-registry/image-registry-77c4ff87fd-tjwtz" Jul 07 17:55:23.104652 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.104455 2583 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13f33577-e89c-4c0f-9264-9a9971e8a17c-registry-certificates\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:55:23.104652 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.104471 2583 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13f33577-e89c-4c0f-9264-9a9971e8a17c-image-registry-private-configuration\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:55:23.104652 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.104485 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87kzf\" (UniqueName: \"kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-kube-api-access-87kzf\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:55:23.104652 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.104500 2583 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-bound-sa-token\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:55:23.104652 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.104514 2583 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13f33577-e89c-4c0f-9264-9a9971e8a17c-registry-tls\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:55:23.104652 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.104528 2583 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13f33577-e89c-4c0f-9264-9a9971e8a17c-installation-pull-secrets\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:55:23.199340 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.199284 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6d876b7659-t4r6h"] Jul 07 17:55:23.204290 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:55:23.204240 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d9f9071_7f2b_4944_88fc_130da6a92ba5.slice/crio-8f34a1ee8e2524a483518f3d57b039acaa6d77be8dd97be30ade4ded592d3536 WatchSource:0}: Error finding container 8f34a1ee8e2524a483518f3d57b039acaa6d77be8dd97be30ade4ded592d3536: Status 404 returned error can't find the container with id 8f34a1ee8e2524a483518f3d57b039acaa6d77be8dd97be30ade4ded592d3536 Jul 07 17:55:23.206578 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.205604 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4cb6eb72-4869-400a-a56d-d712f4c8ff83-trusted-ca\") pod \"image-registry-77c4ff87fd-tjwtz\" (UID: \"4cb6eb72-4869-400a-a56d-d712f4c8ff83\") " pod="openshift-image-registry/image-registry-77c4ff87fd-tjwtz" Jul 07 17:55:23.206578 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.205636 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4cb6eb72-4869-400a-a56d-d712f4c8ff83-image-registry-private-configuration\") pod \"image-registry-77c4ff87fd-tjwtz\" (UID: \"4cb6eb72-4869-400a-a56d-d712f4c8ff83\") " pod="openshift-image-registry/image-registry-77c4ff87fd-tjwtz" Jul 07 17:55:23.206578 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.205662 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cb6eb72-4869-400a-a56d-d712f4c8ff83-registry-tls\") pod \"image-registry-77c4ff87fd-tjwtz\" (UID: \"4cb6eb72-4869-400a-a56d-d712f4c8ff83\") " pod="openshift-image-registry/image-registry-77c4ff87fd-tjwtz" Jul 07 17:55:23.206578 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.205694 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4cb6eb72-4869-400a-a56d-d712f4c8ff83-installation-pull-secrets\") pod \"image-registry-77c4ff87fd-tjwtz\" (UID: \"4cb6eb72-4869-400a-a56d-d712f4c8ff83\") " pod="openshift-image-registry/image-registry-77c4ff87fd-tjwtz" Jul 07 17:55:23.206578 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.205769 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4cb6eb72-4869-400a-a56d-d712f4c8ff83-bound-sa-token\") pod \"image-registry-77c4ff87fd-tjwtz\" (UID: \"4cb6eb72-4869-400a-a56d-d712f4c8ff83\") " pod="openshift-image-registry/image-registry-77c4ff87fd-tjwtz" Jul 07 17:55:23.206578 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.205795 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4cb6eb72-4869-400a-a56d-d712f4c8ff83-registry-certificates\") pod \"image-registry-77c4ff87fd-tjwtz\" (UID: \"4cb6eb72-4869-400a-a56d-d712f4c8ff83\") " pod="openshift-image-registry/image-registry-77c4ff87fd-tjwtz" Jul 07 17:55:23.206578 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.205834 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djtp9\" (UniqueName: \"kubernetes.io/projected/4cb6eb72-4869-400a-a56d-d712f4c8ff83-kube-api-access-djtp9\") pod \"image-registry-77c4ff87fd-tjwtz\" (UID: \"4cb6eb72-4869-400a-a56d-d712f4c8ff83\") " pod="openshift-image-registry/image-registry-77c4ff87fd-tjwtz" Jul 07 17:55:23.206578 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.205885 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4cb6eb72-4869-400a-a56d-d712f4c8ff83-ca-trust-extracted\") pod \"image-registry-77c4ff87fd-tjwtz\" (UID: \"4cb6eb72-4869-400a-a56d-d712f4c8ff83\") " pod="openshift-image-registry/image-registry-77c4ff87fd-tjwtz" Jul 07 17:55:23.206578 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.206207 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4cb6eb72-4869-400a-a56d-d712f4c8ff83-ca-trust-extracted\") pod \"image-registry-77c4ff87fd-tjwtz\" (UID: \"4cb6eb72-4869-400a-a56d-d712f4c8ff83\") " pod="openshift-image-registry/image-registry-77c4ff87fd-tjwtz" Jul 07 17:55:23.207077 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.206832 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4cb6eb72-4869-400a-a56d-d712f4c8ff83-trusted-ca\") pod \"image-registry-77c4ff87fd-tjwtz\" (UID: \"4cb6eb72-4869-400a-a56d-d712f4c8ff83\") " pod="openshift-image-registry/image-registry-77c4ff87fd-tjwtz" Jul 07 17:55:23.209621 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.209229 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4cb6eb72-4869-400a-a56d-d712f4c8ff83-image-registry-private-configuration\") pod \"image-registry-77c4ff87fd-tjwtz\" (UID: \"4cb6eb72-4869-400a-a56d-d712f4c8ff83\") " pod="openshift-image-registry/image-registry-77c4ff87fd-tjwtz" Jul 07 17:55:23.209621 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.209576 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cb6eb72-4869-400a-a56d-d712f4c8ff83-registry-tls\") pod \"image-registry-77c4ff87fd-tjwtz\" (UID: \"4cb6eb72-4869-400a-a56d-d712f4c8ff83\") " pod="openshift-image-registry/image-registry-77c4ff87fd-tjwtz" Jul 07 17:55:23.210863 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.210838 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4cb6eb72-4869-400a-a56d-d712f4c8ff83-installation-pull-secrets\") pod \"image-registry-77c4ff87fd-tjwtz\" (UID: \"4cb6eb72-4869-400a-a56d-d712f4c8ff83\") " pod="openshift-image-registry/image-registry-77c4ff87fd-tjwtz" Jul 07 17:55:23.211198 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.211152 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4cb6eb72-4869-400a-a56d-d712f4c8ff83-registry-certificates\") pod \"image-registry-77c4ff87fd-tjwtz\" (UID: \"4cb6eb72-4869-400a-a56d-d712f4c8ff83\") " pod="openshift-image-registry/image-registry-77c4ff87fd-tjwtz" Jul 07 17:55:23.234724 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.234677 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4cb6eb72-4869-400a-a56d-d712f4c8ff83-bound-sa-token\") pod \"image-registry-77c4ff87fd-tjwtz\" (UID: \"4cb6eb72-4869-400a-a56d-d712f4c8ff83\") " pod="openshift-image-registry/image-registry-77c4ff87fd-tjwtz" Jul 07 17:55:23.234881 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.234859 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djtp9\" (UniqueName: \"kubernetes.io/projected/4cb6eb72-4869-400a-a56d-d712f4c8ff83-kube-api-access-djtp9\") pod \"image-registry-77c4ff87fd-tjwtz\" (UID: \"4cb6eb72-4869-400a-a56d-d712f4c8ff83\") " pod="openshift-image-registry/image-registry-77c4ff87fd-tjwtz" Jul 07 17:55:23.255817 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.255728 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:23.304645 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.304603 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-c44kt\"" Jul 07 17:55:23.312831 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.312793 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77c4ff87fd-tjwtz" Jul 07 17:55:23.395361 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.395330 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:23.464714 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.464682 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77c4ff87fd-tjwtz"] Jul 07 17:55:23.468433 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:55:23.468406 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cb6eb72_4869_400a_a56d_d712f4c8ff83.slice/crio-4980ee22fc3cdd841c34a153e8285f2f2c06ce108992d79ac4bb74e0160b4540 WatchSource:0}: Error finding container 4980ee22fc3cdd841c34a153e8285f2f2c06ce108992d79ac4bb74e0160b4540: Status 404 returned error can't find the container with id 4980ee22fc3cdd841c34a153e8285f2f2c06ce108992d79ac4bb74e0160b4540 Jul 07 17:55:23.851977 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.851939 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xqq9h" event={"ID":"3a1d71c1-6f70-441c-87bb-1f997df1fd5d","Type":"ContainerStarted","Data":"098c67549d369fdf00fd70d96cc051ca864bebc1a7ba23d156270f0fce43f684"} Jul 07 17:55:23.854726 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.854218 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77c4ff87fd-tjwtz" event={"ID":"4cb6eb72-4869-400a-a56d-d712f4c8ff83","Type":"ContainerStarted","Data":"5e4993f85c0f02ea8b1570543af3a79051c5df9262f2ed2578aeb3588879c15d"} Jul 07 17:55:23.854726 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.854268 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77c4ff87fd-tjwtz" event={"ID":"4cb6eb72-4869-400a-a56d-d712f4c8ff83","Type":"ContainerStarted","Data":"4980ee22fc3cdd841c34a153e8285f2f2c06ce108992d79ac4bb74e0160b4540"} Jul 07 17:55:23.854934 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.854776 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-77c4ff87fd-tjwtz" Jul 07 17:55:23.857084 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.857042 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bmdgm" event={"ID":"8a7f4631-8582-4636-832f-e026bb298b27","Type":"ContainerStarted","Data":"13adc08383a70cb7671519828472fdf8fceeaf7730f57851b883094bf4c66a50"} Jul 07 17:55:23.857219 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.857104 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:55:23.858544 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.858518 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6d876b7659-t4r6h" event={"ID":"4d9f9071-7f2b-4944-88fc-130da6a92ba5","Type":"ContainerStarted","Data":"070edd45a40e7d5516da152cc29be73fde12280135548e219c8c0cfdfc84a0d0"} Jul 07 17:55:23.858684 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.858561 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6d876b7659-t4r6h" event={"ID":"4d9f9071-7f2b-4944-88fc-130da6a92ba5","Type":"ContainerStarted","Data":"8f34a1ee8e2524a483518f3d57b039acaa6d77be8dd97be30ade4ded592d3536"} Jul 07 17:55:23.860284 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.860264 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-847hz" event={"ID":"4e09f673-9441-4afe-a258-85b6b5203de8","Type":"ContainerStarted","Data":"0ca5e346a8262a0d8e052413f23fbf9980559beccaf2b7defe0864bc1551f519"} Jul 07 17:55:23.860377 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.860301 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-644cfc977-5rqx8" Jul 07 17:55:23.860377 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.860306 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-847hz" event={"ID":"4e09f673-9441-4afe-a258-85b6b5203de8","Type":"ContainerStarted","Data":"0902be5107d93aaa9efffda76c43d510bc88a27186c55d6a0c8d9112430fb84c"} Jul 07 17:55:23.860456 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.860412 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-847hz" Jul 07 17:55:23.873253 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.873211 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xqq9h" podStartSLOduration=66.243226746 podStartE2EDuration="1m11.87319639s" totalImagesPullingTime="5.629969644s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:55:23.873195423 +0000 UTC m=+72.955799127" watchObservedRunningTime="2026-07-07 17:55:23.87319639 +0000 UTC m=+72.955800085" Jul 07 17:55:23.916195 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.916163 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-644cfc977-5rqx8"] Jul 07 17:55:23.920411 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.920358 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:55:23.922146 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.922115 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-644cfc977-5rqx8"] Jul 07 17:55:23.923061 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.923036 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:55:23.956281 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.956228 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-77c4ff87fd-tjwtz" podStartSLOduration=1.95621172 podStartE2EDuration="1.95621172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:55:23.955574066 +0000 UTC m=+73.038177770" watchObservedRunningTime="2026-07-07 17:55:23.95621172 +0000 UTC m=+73.038815423" Jul 07 17:55:23.980814 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:23.980758 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6d876b7659-t4r6h" podStartSLOduration=33.980741534 podStartE2EDuration="33.980741534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:55:23.979850093 +0000 UTC m=+73.062453794" watchObservedRunningTime="2026-07-07 17:55:23.980741534 +0000 UTC m=+73.063345216" Jul 07 17:55:24.004713 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:24.004662 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-847hz" podStartSLOduration=33.552187765 podStartE2EDuration="37.004643244s" totalImagesPullingTime="3.452455479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:54:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:55:24.004345217 +0000 UTC m=+73.086948918" watchObservedRunningTime="2026-07-07 17:55:24.004643244 +0000 UTC m=+73.087246949" Jul 07 17:55:24.055622 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:24.055556 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bmdgm" podStartSLOduration=66.456536953 podStartE2EDuration="1m12.055535146s" totalImagesPullingTime="5.598998193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:55:24.027795768 +0000 UTC m=+73.110399473" watchObservedRunningTime="2026-07-07 17:55:24.055535146 +0000 UTC m=+73.138138850" Jul 07 17:55:24.864396 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:24.864362 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:55:24.865653 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:24.865631 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6d876b7659-t4r6h" Jul 07 17:55:25.429417 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:25.429384 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13f33577-e89c-4c0f-9264-9a9971e8a17c" path="/var/lib/kubelet/pods/13f33577-e89c-4c0f-9264-9a9971e8a17c/volumes" Jul 07 17:55:27.115234 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:27.115197 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:27.217447 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:27.217415 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:31.648081 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:31.647955 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:31.857164 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:31.857121 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:31.933539 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:31.933459 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-rxn9v"] Jul 07 17:55:31.935210 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:31.935194 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-rxn9v" Jul 07 17:55:31.949467 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:31.949448 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 07 17:55:31.949651 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:31.949633 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-62jsl\"" Jul 07 17:55:31.977549 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:31.977516 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-rxn9v"] Jul 07 17:55:32.020204 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:32.020168 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:32.074225 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:32.074186 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4cc296ee-0400-4cc0-b79a-e21d0cc0929c-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-rxn9v\" (UID: \"4cc296ee-0400-4cc0-b79a-e21d0cc0929c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-rxn9v" Jul 07 17:55:32.175291 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:32.175254 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4cc296ee-0400-4cc0-b79a-e21d0cc0929c-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-rxn9v\" (UID: \"4cc296ee-0400-4cc0-b79a-e21d0cc0929c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-rxn9v" Jul 07 17:55:32.177789 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:32.177766 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4cc296ee-0400-4cc0-b79a-e21d0cc0929c-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-rxn9v\" (UID: \"4cc296ee-0400-4cc0-b79a-e21d0cc0929c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-rxn9v" Jul 07 17:55:32.192476 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:32.192402 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:32.244001 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:32.243971 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-rxn9v" Jul 07 17:55:32.276487 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:32.276453 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44101f7b-1922-44b1-9042-8a5bdc474c87-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89b5k\" (UID: \"44101f7b-1922-44b1-9042-8a5bdc474c87\") " pod="openshift-insights/insights-runtime-extractor-89b5k" Jul 07 17:55:32.278848 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:32.278828 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/44101f7b-1922-44b1-9042-8a5bdc474c87-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89b5k\" (UID: \"44101f7b-1922-44b1-9042-8a5bdc474c87\") " pod="openshift-insights/insights-runtime-extractor-89b5k" Jul 07 17:55:32.313862 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:32.313818 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:32.386511 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:32.386480 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-rxn9v"] Jul 07 17:55:32.391925 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:55:32.391877 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc296ee_0400_4cc0_b79a_e21d0cc0929c.slice/crio-1f2027a1a9de7ac9729cfe2d9f640fcd79219e1382778ef675c24ce1903553d3 WatchSource:0}: Error finding container 1f2027a1a9de7ac9729cfe2d9f640fcd79219e1382778ef675c24ce1903553d3: Status 404 returned error can't find the container with id 1f2027a1a9de7ac9729cfe2d9f640fcd79219e1382778ef675c24ce1903553d3 Jul 07 17:55:32.577835 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:32.577805 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5jtj4\"" Jul 07 17:55:32.585255 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:32.585226 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-89b5k" Jul 07 17:55:32.727923 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:32.727877 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-89b5k"] Jul 07 17:55:32.732703 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:55:32.732674 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44101f7b_1922_44b1_9042_8a5bdc474c87.slice/crio-3081977962bfe99fa907e2404803a5760cb347e24bf2ef737a30ab765bed251e WatchSource:0}: Error finding container 3081977962bfe99fa907e2404803a5760cb347e24bf2ef737a30ab765bed251e: Status 404 returned error can't find the container with id 3081977962bfe99fa907e2404803a5760cb347e24bf2ef737a30ab765bed251e Jul 07 17:55:32.893648 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:32.893546 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-89b5k" event={"ID":"44101f7b-1922-44b1-9042-8a5bdc474c87","Type":"ContainerStarted","Data":"3081977962bfe99fa907e2404803a5760cb347e24bf2ef737a30ab765bed251e"} Jul 07 17:55:32.894981 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:32.894945 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-rxn9v" event={"ID":"4cc296ee-0400-4cc0-b79a-e21d0cc0929c","Type":"ContainerStarted","Data":"1f2027a1a9de7ac9729cfe2d9f640fcd79219e1382778ef675c24ce1903553d3"} Jul 07 17:55:33.866457 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:33.866427 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-847hz" Jul 07 17:55:33.900284 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:33.900224 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-rxn9v" event={"ID":"4cc296ee-0400-4cc0-b79a-e21d0cc0929c","Type":"ContainerStarted","Data":"d124e6c159f704ee77c1391bd65205762e87774856e9a5e4f9a94fac8f0db699"} Jul 07 17:55:33.900486 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:33.900443 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-rxn9v" Jul 07 17:55:33.906090 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:33.906060 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-rxn9v" Jul 07 17:55:33.992402 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:33.992350 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-rxn9v" podStartSLOduration=1.84705662 podStartE2EDuration="2.992332074s" totalImagesPullingTime="1.145275454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:55:33.946187444 +0000 UTC m=+83.028791148" watchObservedRunningTime="2026-07-07 17:55:33.992332074 +0000 UTC m=+83.074935768" Jul 07 17:55:34.946094 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:34.946038 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:35.057362 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:35.057324 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:35.159475 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:35.158881 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:36.423779 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:36.423743 2583 scope.go:117] "RemoveContainer" containerID="e6b1245a376909f6b71c28cdd2f927c690e644ffe4e5422b0b5b82ad4136de58" Jul 07 17:55:36.424223 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:36.423965 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-wqfm4_openshift-console-operator(997df85c-7632-47b5-a8a6-1c37f3346251)\"" pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" podUID="997df85c-7632-47b5-a8a6-1c37f3346251" Jul 07 17:55:36.913141 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:36.913106 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-89b5k" event={"ID":"44101f7b-1922-44b1-9042-8a5bdc474c87","Type":"ContainerStarted","Data":"18386c9fda7975150833116d8f3595ec4d95147908b9ddc71bf7964ff64f3e09"} Jul 07 17:55:37.919726 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:37.919682 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-89b5k" event={"ID":"44101f7b-1922-44b1-9042-8a5bdc474c87","Type":"ContainerStarted","Data":"244445fcabbf60abdc8a0176e74d8dfe4b842280deb168ecc7c4769c5033458a"} Jul 07 17:55:38.595210 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:38.595139 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:40.937708 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:40.937670 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-89b5k" event={"ID":"44101f7b-1922-44b1-9042-8a5bdc474c87","Type":"ContainerStarted","Data":"59a170dd038ee7c7f9f90a03d228cd122c2edaa660805926ce35c719a0476816"} Jul 07 17:55:40.965595 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:40.965535 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-89b5k" podStartSLOduration=33.92749321 podStartE2EDuration="40.965518254s" totalImagesPullingTime="7.038025044s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 17:55:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:55:40.964508892 +0000 UTC m=+90.047112597" watchObservedRunningTime="2026-07-07 17:55:40.965518254 +0000 UTC m=+90.048121955" Jul 07 17:55:41.263063 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.262983 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:41.538836 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.538749 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:41.613881 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.613843 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gws7k"] Jul 07 17:55:41.617325 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.617304 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gws7k" Jul 07 17:55:41.620936 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.620889 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 07 17:55:41.621458 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.621440 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 07 17:55:41.621543 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.621460 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qjnn2\"" Jul 07 17:55:41.621750 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.621734 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 07 17:55:41.621892 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.621875 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tkg2m"] Jul 07 17:55:41.622286 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.622257 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 07 17:55:41.622564 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.622546 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 07 17:55:41.622812 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.622797 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 07 17:55:41.625079 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.625061 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tkg2m" Jul 07 17:55:41.628548 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.628531 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 07 17:55:41.628642 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.628533 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 07 17:55:41.633037 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.633018 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-dxtps\"" Jul 07 17:55:41.635793 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.635778 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 07 17:55:41.648561 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.648526 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-tkg2m\" (UID: \"ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tkg2m" Jul 07 17:55:41.648677 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.648566 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6dbc6247-19f3-4e77-b809-18b641f4bdf9-root\") pod \"node-exporter-gws7k\" (UID: \"6dbc6247-19f3-4e77-b809-18b641f4bdf9\") " pod="openshift-monitoring/node-exporter-gws7k" Jul 07 17:55:41.648677 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.648588 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6dbc6247-19f3-4e77-b809-18b641f4bdf9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gws7k\" (UID: \"6dbc6247-19f3-4e77-b809-18b641f4bdf9\") " pod="openshift-monitoring/node-exporter-gws7k" Jul 07 17:55:41.648677 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.648610 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-tkg2m\" (UID: \"ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tkg2m" Jul 07 17:55:41.648677 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.648635 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6dbc6247-19f3-4e77-b809-18b641f4bdf9-node-exporter-textfile\") pod \"node-exporter-gws7k\" (UID: \"6dbc6247-19f3-4e77-b809-18b641f4bdf9\") " pod="openshift-monitoring/node-exporter-gws7k" Jul 07 17:55:41.648677 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.648650 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-tkg2m\" (UID: \"ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tkg2m" Jul 07 17:55:41.648677 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.648674 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6dbc6247-19f3-4e77-b809-18b641f4bdf9-node-exporter-accelerators-collector-config\") pod \"node-exporter-gws7k\" (UID: \"6dbc6247-19f3-4e77-b809-18b641f4bdf9\") " pod="openshift-monitoring/node-exporter-gws7k" Jul 07 17:55:41.648857 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.648706 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-tkg2m\" (UID: \"ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tkg2m" Jul 07 17:55:41.648857 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.648745 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdc4l\" (UniqueName: \"kubernetes.io/projected/6dbc6247-19f3-4e77-b809-18b641f4bdf9-kube-api-access-jdc4l\") pod \"node-exporter-gws7k\" (UID: \"6dbc6247-19f3-4e77-b809-18b641f4bdf9\") " pod="openshift-monitoring/node-exporter-gws7k" Jul 07 17:55:41.648857 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.648796 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dbc6247-19f3-4e77-b809-18b641f4bdf9-metrics-client-ca\") pod \"node-exporter-gws7k\" (UID: \"6dbc6247-19f3-4e77-b809-18b641f4bdf9\") " pod="openshift-monitoring/node-exporter-gws7k" Jul 07 17:55:41.648857 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.648833 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6dbc6247-19f3-4e77-b809-18b641f4bdf9-sys\") pod \"node-exporter-gws7k\" (UID: \"6dbc6247-19f3-4e77-b809-18b641f4bdf9\") " pod="openshift-monitoring/node-exporter-gws7k" Jul 07 17:55:41.648857 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.648852 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvlcp\" (UniqueName: \"kubernetes.io/projected/ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1-kube-api-access-bvlcp\") pod \"kube-state-metrics-7c9dbb8b7d-tkg2m\" (UID: \"ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tkg2m" Jul 07 17:55:41.649019 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.648883 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6dbc6247-19f3-4e77-b809-18b641f4bdf9-node-exporter-wtmp\") pod \"node-exporter-gws7k\" (UID: \"6dbc6247-19f3-4e77-b809-18b641f4bdf9\") " pod="openshift-monitoring/node-exporter-gws7k" Jul 07 17:55:41.649019 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.648966 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-tkg2m\" (UID: \"ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tkg2m" Jul 07 17:55:41.649019 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.648983 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6dbc6247-19f3-4e77-b809-18b641f4bdf9-node-exporter-tls\") pod \"node-exporter-gws7k\" (UID: \"6dbc6247-19f3-4e77-b809-18b641f4bdf9\") " pod="openshift-monitoring/node-exporter-gws7k" Jul 07 17:55:41.652472 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.652448 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tkg2m"] Jul 07 17:55:41.666781 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.666758 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:41.750187 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.750157 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6dbc6247-19f3-4e77-b809-18b641f4bdf9-node-exporter-wtmp\") pod \"node-exporter-gws7k\" (UID: \"6dbc6247-19f3-4e77-b809-18b641f4bdf9\") " pod="openshift-monitoring/node-exporter-gws7k" Jul 07 17:55:41.750187 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.750200 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-tkg2m\" (UID: \"ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tkg2m" Jul 07 17:55:41.750438 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.750221 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6dbc6247-19f3-4e77-b809-18b641f4bdf9-node-exporter-tls\") pod \"node-exporter-gws7k\" (UID: \"6dbc6247-19f3-4e77-b809-18b641f4bdf9\") " pod="openshift-monitoring/node-exporter-gws7k" Jul 07 17:55:41.750438 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.750244 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-tkg2m\" (UID: \"ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tkg2m" Jul 07 17:55:41.750438 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.750274 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6dbc6247-19f3-4e77-b809-18b641f4bdf9-root\") pod \"node-exporter-gws7k\" (UID: \"6dbc6247-19f3-4e77-b809-18b641f4bdf9\") " pod="openshift-monitoring/node-exporter-gws7k" Jul 07 17:55:41.750438 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.750313 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6dbc6247-19f3-4e77-b809-18b641f4bdf9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gws7k\" (UID: \"6dbc6247-19f3-4e77-b809-18b641f4bdf9\") " pod="openshift-monitoring/node-exporter-gws7k" Jul 07 17:55:41.750438 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.750338 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-tkg2m\" (UID: \"ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tkg2m" Jul 07 17:55:41.750438 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.750367 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6dbc6247-19f3-4e77-b809-18b641f4bdf9-node-exporter-textfile\") pod \"node-exporter-gws7k\" (UID: \"6dbc6247-19f3-4e77-b809-18b641f4bdf9\") " pod="openshift-monitoring/node-exporter-gws7k" Jul 07 17:55:41.750438 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.750391 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-tkg2m\" (UID: \"ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tkg2m" Jul 07 17:55:41.750438 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.750412 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6dbc6247-19f3-4e77-b809-18b641f4bdf9-root\") pod \"node-exporter-gws7k\" (UID: \"6dbc6247-19f3-4e77-b809-18b641f4bdf9\") " pod="openshift-monitoring/node-exporter-gws7k" Jul 07 17:55:41.750438 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.750428 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6dbc6247-19f3-4e77-b809-18b641f4bdf9-node-exporter-accelerators-collector-config\") pod \"node-exporter-gws7k\" (UID: \"6dbc6247-19f3-4e77-b809-18b641f4bdf9\") " pod="openshift-monitoring/node-exporter-gws7k" Jul 07 17:55:41.750861 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.750387 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6dbc6247-19f3-4e77-b809-18b641f4bdf9-node-exporter-wtmp\") pod \"node-exporter-gws7k\" (UID: \"6dbc6247-19f3-4e77-b809-18b641f4bdf9\") " pod="openshift-monitoring/node-exporter-gws7k" Jul 07 17:55:41.750861 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.750449 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-tkg2m\" (UID: \"ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tkg2m" Jul 07 17:55:41.750861 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.750491 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdc4l\" (UniqueName: \"kubernetes.io/projected/6dbc6247-19f3-4e77-b809-18b641f4bdf9-kube-api-access-jdc4l\") pod \"node-exporter-gws7k\" (UID: \"6dbc6247-19f3-4e77-b809-18b641f4bdf9\") " pod="openshift-monitoring/node-exporter-gws7k" Jul 07 17:55:41.750861 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.750526 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dbc6247-19f3-4e77-b809-18b641f4bdf9-metrics-client-ca\") pod \"node-exporter-gws7k\" (UID: \"6dbc6247-19f3-4e77-b809-18b641f4bdf9\") " pod="openshift-monitoring/node-exporter-gws7k" Jul 07 17:55:41.750861 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.750548 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6dbc6247-19f3-4e77-b809-18b641f4bdf9-sys\") pod \"node-exporter-gws7k\" (UID: \"6dbc6247-19f3-4e77-b809-18b641f4bdf9\") " pod="openshift-monitoring/node-exporter-gws7k" Jul 07 17:55:41.750861 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.750574 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvlcp\" (UniqueName: \"kubernetes.io/projected/ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1-kube-api-access-bvlcp\") pod \"kube-state-metrics-7c9dbb8b7d-tkg2m\" (UID: \"ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tkg2m" Jul 07 17:55:41.751177 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.750990 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6dbc6247-19f3-4e77-b809-18b641f4bdf9-node-exporter-textfile\") pod \"node-exporter-gws7k\" (UID: \"6dbc6247-19f3-4e77-b809-18b641f4bdf9\") " pod="openshift-monitoring/node-exporter-gws7k" Jul 07 17:55:41.751667 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.751442 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-tkg2m\" (UID: \"ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tkg2m" Jul 07 17:55:41.751667 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.751449 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-tkg2m\" (UID: \"ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tkg2m" Jul 07 17:55:41.751667 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.751491 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6dbc6247-19f3-4e77-b809-18b641f4bdf9-sys\") pod \"node-exporter-gws7k\" (UID: \"6dbc6247-19f3-4e77-b809-18b641f4bdf9\") " pod="openshift-monitoring/node-exporter-gws7k" Jul 07 17:55:41.751667 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.751566 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6dbc6247-19f3-4e77-b809-18b641f4bdf9-node-exporter-accelerators-collector-config\") pod \"node-exporter-gws7k\" (UID: \"6dbc6247-19f3-4e77-b809-18b641f4bdf9\") " pod="openshift-monitoring/node-exporter-gws7k" Jul 07 17:55:41.751972 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.751699 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dbc6247-19f3-4e77-b809-18b641f4bdf9-metrics-client-ca\") pod \"node-exporter-gws7k\" (UID: \"6dbc6247-19f3-4e77-b809-18b641f4bdf9\") " pod="openshift-monitoring/node-exporter-gws7k" Jul 07 17:55:41.752032 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.752002 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-tkg2m\" (UID: \"ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tkg2m" Jul 07 17:55:41.753120 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.753089 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6dbc6247-19f3-4e77-b809-18b641f4bdf9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gws7k\" (UID: \"6dbc6247-19f3-4e77-b809-18b641f4bdf9\") " pod="openshift-monitoring/node-exporter-gws7k" Jul 07 17:55:41.753211 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.753125 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6dbc6247-19f3-4e77-b809-18b641f4bdf9-node-exporter-tls\") pod \"node-exporter-gws7k\" (UID: \"6dbc6247-19f3-4e77-b809-18b641f4bdf9\") " pod="openshift-monitoring/node-exporter-gws7k" Jul 07 17:55:41.753211 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.753192 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-tkg2m\" (UID: \"ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tkg2m" Jul 07 17:55:41.753473 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.753450 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-tkg2m\" (UID: \"ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tkg2m" Jul 07 17:55:41.761774 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.761749 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvlcp\" (UniqueName: \"kubernetes.io/projected/ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1-kube-api-access-bvlcp\") pod \"kube-state-metrics-7c9dbb8b7d-tkg2m\" (UID: \"ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tkg2m" Jul 07 17:55:41.762130 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.762111 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdc4l\" (UniqueName: \"kubernetes.io/projected/6dbc6247-19f3-4e77-b809-18b641f4bdf9-kube-api-access-jdc4l\") pod \"node-exporter-gws7k\" (UID: \"6dbc6247-19f3-4e77-b809-18b641f4bdf9\") " pod="openshift-monitoring/node-exporter-gws7k" Jul 07 17:55:41.795153 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.795123 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:41.927496 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.927456 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gws7k" Jul 07 17:55:41.933852 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.933823 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tkg2m" Jul 07 17:55:41.940098 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:55:41.939987 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dbc6247_19f3_4e77_b809_18b641f4bdf9.slice/crio-543acfbf63d6bf0be62dda6dc9c0c46c0af6ae587bb9f694d865091d1f2e4b94 WatchSource:0}: Error finding container 543acfbf63d6bf0be62dda6dc9c0c46c0af6ae587bb9f694d865091d1f2e4b94: Status 404 returned error can't find the container with id 543acfbf63d6bf0be62dda6dc9c0c46c0af6ae587bb9f694d865091d1f2e4b94 Jul 07 17:55:41.946782 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:41.946763 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:42.039166 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.039128 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:42.077655 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.077555 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tkg2m"] Jul 07 17:55:42.082501 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:55:42.082467 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded8ec45f_7e9f_4b47_8f7d_09813c73d4d1.slice/crio-1cea5366ccb0b3df2f6b2bb22bed1a4722963d23e086d9dc898a9423359ad44e WatchSource:0}: Error finding container 1cea5366ccb0b3df2f6b2bb22bed1a4722963d23e086d9dc898a9423359ad44e: Status 404 returned error can't find the container with id 1cea5366ccb0b3df2f6b2bb22bed1a4722963d23e086d9dc898a9423359ad44e Jul 07 17:55:42.144221 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.144138 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:42.283437 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.283403 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:42.401612 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.401495 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:42.579453 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.579396 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:42.727597 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.727506 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:42.744598 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.744560 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 17:55:42.748823 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.748794 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.756570 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.756542 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 07 17:55:42.756785 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.756762 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 07 17:55:42.756865 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.756815 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-m2487\"" Jul 07 17:55:42.757072 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.757057 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 07 17:55:42.757165 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.757112 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 07 17:55:42.757165 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.757127 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 07 17:55:42.757260 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.757250 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 07 17:55:42.758601 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.758577 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-config-out\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.758716 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.758618 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.758716 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.758643 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.758716 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.758664 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.758716 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.758685 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.758933 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.758765 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.758933 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.758792 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.758933 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.758814 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-config-volume\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.758933 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.758828 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-web-config\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.758933 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.758870 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv2xr\" (UniqueName: \"kubernetes.io/projected/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-kube-api-access-wv2xr\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.758933 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.758927 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.759187 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.758954 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.759187 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.758994 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.760574 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.760557 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 07 17:55:42.764188 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.764168 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 07 17:55:42.764286 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.764183 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 07 17:55:42.773420 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.773393 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 17:55:42.859978 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.859935 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.860174 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.859988 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.860174 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.860035 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.860174 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.860052 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.860174 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.860077 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-config-volume\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.860174 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.860102 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-web-config\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.860174 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.860136 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv2xr\" (UniqueName: \"kubernetes.io/projected/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-kube-api-access-wv2xr\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.860174 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.860155 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.860583 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.860178 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.860583 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.860237 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.860583 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.860262 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-config-out\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.860583 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.860281 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.860583 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.860315 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.862149 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:42.861030 2583 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 07 17:55:42.862149 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:42.861107 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-secret-alertmanager-main-tls podName:d0a24e05-c4bb-4beb-b46e-ec0e58d0832e nodeName:}" failed. No retries permitted until 2026-07-07 17:55:43.361083892 +0000 UTC m=+92.443687596 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "d0a24e05-c4bb-4beb-b46e-ec0e58d0832e") : secret "alertmanager-main-tls" not found Jul 07 17:55:42.862149 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.861888 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.862149 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.861883 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.862149 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.862010 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.866647 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.866618 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.869827 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.869641 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.869827 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.869645 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.869827 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.869782 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.869827 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.869804 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-config-out\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.870312 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.870287 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.871329 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.871306 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-config-volume\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.872193 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.872172 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-web-config\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.875827 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.874343 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv2xr\" (UniqueName: \"kubernetes.io/projected/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-kube-api-access-wv2xr\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:42.923346 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.923302 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:42.947977 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.947937 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gws7k" event={"ID":"6dbc6247-19f3-4e77-b809-18b641f4bdf9","Type":"ContainerStarted","Data":"543acfbf63d6bf0be62dda6dc9c0c46c0af6ae587bb9f694d865091d1f2e4b94"} Jul 07 17:55:42.949440 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:42.949410 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tkg2m" event={"ID":"ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1","Type":"ContainerStarted","Data":"1cea5366ccb0b3df2f6b2bb22bed1a4722963d23e086d9dc898a9423359ad44e"} Jul 07 17:55:43.016459 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.016355 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:43.364111 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.364084 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:43.364244 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:43.364230 2583 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 07 17:55:43.364308 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:55:43.364299 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-secret-alertmanager-main-tls podName:d0a24e05-c4bb-4beb-b46e-ec0e58d0832e nodeName:}" failed. No retries permitted until 2026-07-07 17:55:44.364280684 +0000 UTC m=+93.446884367 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "d0a24e05-c4bb-4beb-b46e-ec0e58d0832e") : secret "alertmanager-main-tls" not found Jul 07 17:55:43.577476 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.577450 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-568b77c849-rc4mb"] Jul 07 17:55:43.581149 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.581128 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" Jul 07 17:55:43.588939 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.588878 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 07 17:55:43.589069 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.588939 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 07 17:55:43.589173 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.588885 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 07 17:55:43.589273 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.589245 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 07 17:55:43.589332 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.589288 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 07 17:55:43.589444 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.589401 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-f08hp28f1pemm\"" Jul 07 17:55:43.589545 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.589480 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-8gfds\"" Jul 07 17:55:43.606055 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.605986 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-568b77c849-rc4mb"] Jul 07 17:55:43.667555 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.667507 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4a3bebfc-1f1e-4bb2-a02a-7174390c38b5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-568b77c849-rc4mb\" (UID: \"4a3bebfc-1f1e-4bb2-a02a-7174390c38b5\") " pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" Jul 07 17:55:43.667555 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.667556 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4a3bebfc-1f1e-4bb2-a02a-7174390c38b5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-568b77c849-rc4mb\" (UID: \"4a3bebfc-1f1e-4bb2-a02a-7174390c38b5\") " pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" Jul 07 17:55:43.667784 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.667640 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a3bebfc-1f1e-4bb2-a02a-7174390c38b5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-568b77c849-rc4mb\" (UID: \"4a3bebfc-1f1e-4bb2-a02a-7174390c38b5\") " pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" Jul 07 17:55:43.667784 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.667699 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4a3bebfc-1f1e-4bb2-a02a-7174390c38b5-secret-grpc-tls\") pod \"thanos-querier-568b77c849-rc4mb\" (UID: \"4a3bebfc-1f1e-4bb2-a02a-7174390c38b5\") " pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" Jul 07 17:55:43.667784 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.667768 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn5xm\" (UniqueName: \"kubernetes.io/projected/4a3bebfc-1f1e-4bb2-a02a-7174390c38b5-kube-api-access-fn5xm\") pod \"thanos-querier-568b77c849-rc4mb\" (UID: \"4a3bebfc-1f1e-4bb2-a02a-7174390c38b5\") " pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" Jul 07 17:55:43.667977 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.667807 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a3bebfc-1f1e-4bb2-a02a-7174390c38b5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-568b77c849-rc4mb\" (UID: \"4a3bebfc-1f1e-4bb2-a02a-7174390c38b5\") " pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" Jul 07 17:55:43.667977 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.667862 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4a3bebfc-1f1e-4bb2-a02a-7174390c38b5-secret-thanos-querier-tls\") pod \"thanos-querier-568b77c849-rc4mb\" (UID: \"4a3bebfc-1f1e-4bb2-a02a-7174390c38b5\") " pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" Jul 07 17:55:43.667977 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.667928 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a3bebfc-1f1e-4bb2-a02a-7174390c38b5-metrics-client-ca\") pod \"thanos-querier-568b77c849-rc4mb\" (UID: \"4a3bebfc-1f1e-4bb2-a02a-7174390c38b5\") " pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" Jul 07 17:55:43.707170 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.707129 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:43.768500 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.768466 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a3bebfc-1f1e-4bb2-a02a-7174390c38b5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-568b77c849-rc4mb\" (UID: \"4a3bebfc-1f1e-4bb2-a02a-7174390c38b5\") " pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" Jul 07 17:55:43.768662 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.768526 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4a3bebfc-1f1e-4bb2-a02a-7174390c38b5-secret-thanos-querier-tls\") pod \"thanos-querier-568b77c849-rc4mb\" (UID: \"4a3bebfc-1f1e-4bb2-a02a-7174390c38b5\") " pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" Jul 07 17:55:43.768662 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.768564 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a3bebfc-1f1e-4bb2-a02a-7174390c38b5-metrics-client-ca\") pod \"thanos-querier-568b77c849-rc4mb\" (UID: \"4a3bebfc-1f1e-4bb2-a02a-7174390c38b5\") " pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" Jul 07 17:55:43.768662 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.768618 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4a3bebfc-1f1e-4bb2-a02a-7174390c38b5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-568b77c849-rc4mb\" (UID: \"4a3bebfc-1f1e-4bb2-a02a-7174390c38b5\") " pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" Jul 07 17:55:43.768662 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.768642 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4a3bebfc-1f1e-4bb2-a02a-7174390c38b5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-568b77c849-rc4mb\" (UID: \"4a3bebfc-1f1e-4bb2-a02a-7174390c38b5\") " pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" Jul 07 17:55:43.768860 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.768757 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a3bebfc-1f1e-4bb2-a02a-7174390c38b5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-568b77c849-rc4mb\" (UID: \"4a3bebfc-1f1e-4bb2-a02a-7174390c38b5\") " pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" Jul 07 17:55:43.768860 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.768808 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4a3bebfc-1f1e-4bb2-a02a-7174390c38b5-secret-grpc-tls\") pod \"thanos-querier-568b77c849-rc4mb\" (UID: \"4a3bebfc-1f1e-4bb2-a02a-7174390c38b5\") " pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" Jul 07 17:55:43.768974 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.768870 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn5xm\" (UniqueName: \"kubernetes.io/projected/4a3bebfc-1f1e-4bb2-a02a-7174390c38b5-kube-api-access-fn5xm\") pod \"thanos-querier-568b77c849-rc4mb\" (UID: \"4a3bebfc-1f1e-4bb2-a02a-7174390c38b5\") " pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" Jul 07 17:55:43.769445 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.769419 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a3bebfc-1f1e-4bb2-a02a-7174390c38b5-metrics-client-ca\") pod \"thanos-querier-568b77c849-rc4mb\" (UID: \"4a3bebfc-1f1e-4bb2-a02a-7174390c38b5\") " pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" Jul 07 17:55:43.771777 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.771727 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4a3bebfc-1f1e-4bb2-a02a-7174390c38b5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-568b77c849-rc4mb\" (UID: \"4a3bebfc-1f1e-4bb2-a02a-7174390c38b5\") " pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" Jul 07 17:55:43.771777 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.771727 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4a3bebfc-1f1e-4bb2-a02a-7174390c38b5-secret-thanos-querier-tls\") pod \"thanos-querier-568b77c849-rc4mb\" (UID: \"4a3bebfc-1f1e-4bb2-a02a-7174390c38b5\") " pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" Jul 07 17:55:43.772028 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.772010 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a3bebfc-1f1e-4bb2-a02a-7174390c38b5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-568b77c849-rc4mb\" (UID: \"4a3bebfc-1f1e-4bb2-a02a-7174390c38b5\") " pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" Jul 07 17:55:43.772091 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.772043 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4a3bebfc-1f1e-4bb2-a02a-7174390c38b5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-568b77c849-rc4mb\" (UID: \"4a3bebfc-1f1e-4bb2-a02a-7174390c38b5\") " pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" Jul 07 17:55:43.775939 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.772575 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a3bebfc-1f1e-4bb2-a02a-7174390c38b5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-568b77c849-rc4mb\" (UID: \"4a3bebfc-1f1e-4bb2-a02a-7174390c38b5\") " pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" Jul 07 17:55:43.775939 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.772614 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4a3bebfc-1f1e-4bb2-a02a-7174390c38b5-secret-grpc-tls\") pod \"thanos-querier-568b77c849-rc4mb\" (UID: \"4a3bebfc-1f1e-4bb2-a02a-7174390c38b5\") " pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" Jul 07 17:55:43.779468 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.779446 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn5xm\" (UniqueName: \"kubernetes.io/projected/4a3bebfc-1f1e-4bb2-a02a-7174390c38b5-kube-api-access-fn5xm\") pod \"thanos-querier-568b77c849-rc4mb\" (UID: \"4a3bebfc-1f1e-4bb2-a02a-7174390c38b5\") " pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" Jul 07 17:55:43.894208 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.893821 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" Jul 07 17:55:43.957407 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.957160 2583 generic.go:356] "Generic (PLEG): container finished" podID="6dbc6247-19f3-4e77-b809-18b641f4bdf9" containerID="25ab9d7a5953fed49cff340bc1fc448b3cafd193f963f0b9ee721d074bb87990" exitCode=0 Jul 07 17:55:43.957407 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.957261 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gws7k" event={"ID":"6dbc6247-19f3-4e77-b809-18b641f4bdf9","Type":"ContainerDied","Data":"25ab9d7a5953fed49cff340bc1fc448b3cafd193f963f0b9ee721d074bb87990"} Jul 07 17:55:43.960605 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.959812 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tkg2m" event={"ID":"ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1","Type":"ContainerStarted","Data":"0db339d2e54359adee4d82a29e26cf9943b9298e940116a0c22020733dce66fb"} Jul 07 17:55:43.960605 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.959868 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tkg2m" event={"ID":"ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1","Type":"ContainerStarted","Data":"980b05003eac12bae608f9e339243d1e521a5c76bd302e83e2af36c324ae4f8a"} Jul 07 17:55:43.960605 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:43.959893 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tkg2m" event={"ID":"ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1","Type":"ContainerStarted","Data":"0fe7535a76db60d35b189e23f32d4026bce280f9de9b693a903eaff8bce6043f"} Jul 07 17:55:44.007071 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:44.006862 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tkg2m" podStartSLOduration=1.693428098 podStartE2EDuration="3.006845516s" totalImagesPullingTime="1.313417418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:55:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:55:44.005646196 +0000 UTC m=+93.088249903" watchObservedRunningTime="2026-07-07 17:55:44.006845516 +0000 UTC m=+93.089449219" Jul 07 17:55:44.043560 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:44.043531 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-568b77c849-rc4mb"] Jul 07 17:55:44.045871 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:55:44.045843 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a3bebfc_1f1e_4bb2_a02a_7174390c38b5.slice/crio-616543c74218b8b220a1e11dede31b95984cd280fd42178796d4a73cc908b171 WatchSource:0}: Error finding container 616543c74218b8b220a1e11dede31b95984cd280fd42178796d4a73cc908b171: Status 404 returned error can't find the container with id 616543c74218b8b220a1e11dede31b95984cd280fd42178796d4a73cc908b171 Jul 07 17:55:44.374794 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:44.374749 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:44.377425 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:44.377396 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:44.566666 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:44.566632 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:55:44.616180 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:44.616138 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:44.767260 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:44.767217 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:44.795508 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:44.795476 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 17:55:44.797363 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:55:44.797333 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a24e05_c4bb_4beb_b46e_ec0e58d0832e.slice/crio-862a59350aa9931a6a07b8ff33b2c32b243a61567dc6ab1e1e041ef4ff9a2d7f WatchSource:0}: Error finding container 862a59350aa9931a6a07b8ff33b2c32b243a61567dc6ab1e1e041ef4ff9a2d7f: Status 404 returned error can't find the container with id 862a59350aa9931a6a07b8ff33b2c32b243a61567dc6ab1e1e041ef4ff9a2d7f Jul 07 17:55:44.934802 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:44.934713 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:44.965109 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:44.965065 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" event={"ID":"4a3bebfc-1f1e-4bb2-a02a-7174390c38b5","Type":"ContainerStarted","Data":"616543c74218b8b220a1e11dede31b95984cd280fd42178796d4a73cc908b171"} Jul 07 17:55:44.967712 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:44.967680 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gws7k" event={"ID":"6dbc6247-19f3-4e77-b809-18b641f4bdf9","Type":"ContainerStarted","Data":"74b166b2c45e5257b66750787c269ca4ec4c3b79f09fe988e55a57bd871d6112"} Jul 07 17:55:44.967849 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:44.967745 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gws7k" event={"ID":"6dbc6247-19f3-4e77-b809-18b641f4bdf9","Type":"ContainerStarted","Data":"d9e46807c28fb57630be3621885a6d7ab594af14296a7b1598ee550530432a3c"} Jul 07 17:55:44.969113 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:44.969072 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e","Type":"ContainerStarted","Data":"862a59350aa9931a6a07b8ff33b2c32b243a61567dc6ab1e1e041ef4ff9a2d7f"} Jul 07 17:55:44.995031 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:44.994974 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gws7k" podStartSLOduration=2.509393797 podStartE2EDuration="3.994954359s" totalImagesPullingTime="1.408601253s" totalInitContainerRuntime="76.959309ms" isStatefulPod=true podCreationTimestamp="2026-07-07 17:55:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:55:44.993720355 +0000 UTC m=+94.076324061" watchObservedRunningTime="2026-07-07 17:55:44.994954359 +0000 UTC m=+94.077558061" Jul 07 17:55:45.211262 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:45.211181 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:45.872776 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:45.872742 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-77c4ff87fd-tjwtz" Jul 07 17:55:46.061816 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.061742 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-74cbbd4d86-rjp57"] Jul 07 17:55:46.065185 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.065167 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-74cbbd4d86-rjp57" Jul 07 17:55:46.069195 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.069177 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-91q0usiod2vi1\"" Jul 07 17:55:46.069285 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.069217 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 07 17:55:46.069738 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.069722 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 07 17:55:46.069821 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.069740 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-sjdst\"" Jul 07 17:55:46.070002 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.069988 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 07 17:55:46.070219 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.070206 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 07 17:55:46.084990 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.084968 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-74cbbd4d86-rjp57"] Jul 07 17:55:46.092191 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.092167 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e8864ce-4a84-4e3c-b838-a1a489470e5a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-74cbbd4d86-rjp57\" (UID: \"9e8864ce-4a84-4e3c-b838-a1a489470e5a\") " pod="openshift-monitoring/metrics-server-74cbbd4d86-rjp57" Jul 07 17:55:46.092302 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.092282 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9e8864ce-4a84-4e3c-b838-a1a489470e5a-audit-log\") pod \"metrics-server-74cbbd4d86-rjp57\" (UID: \"9e8864ce-4a84-4e3c-b838-a1a489470e5a\") " pod="openshift-monitoring/metrics-server-74cbbd4d86-rjp57" Jul 07 17:55:46.092378 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.092332 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9e8864ce-4a84-4e3c-b838-a1a489470e5a-client-ca-bundle\") pod \"metrics-server-74cbbd4d86-rjp57\" (UID: \"9e8864ce-4a84-4e3c-b838-a1a489470e5a\") " pod="openshift-monitoring/metrics-server-74cbbd4d86-rjp57" Jul 07 17:55:46.092441 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.092373 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9e8864ce-4a84-4e3c-b838-a1a489470e5a-secret-metrics-server-client-certs\") pod \"metrics-server-74cbbd4d86-rjp57\" (UID: \"9e8864ce-4a84-4e3c-b838-a1a489470e5a\") " pod="openshift-monitoring/metrics-server-74cbbd4d86-rjp57" Jul 07 17:55:46.092441 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.092401 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf8z6\" (UniqueName: \"kubernetes.io/projected/9e8864ce-4a84-4e3c-b838-a1a489470e5a-kube-api-access-cf8z6\") pod \"metrics-server-74cbbd4d86-rjp57\" (UID: \"9e8864ce-4a84-4e3c-b838-a1a489470e5a\") " pod="openshift-monitoring/metrics-server-74cbbd4d86-rjp57" Jul 07 17:55:46.092592 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.092566 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9e8864ce-4a84-4e3c-b838-a1a489470e5a-metrics-server-audit-profiles\") pod \"metrics-server-74cbbd4d86-rjp57\" (UID: \"9e8864ce-4a84-4e3c-b838-a1a489470e5a\") " pod="openshift-monitoring/metrics-server-74cbbd4d86-rjp57" Jul 07 17:55:46.092700 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.092681 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9e8864ce-4a84-4e3c-b838-a1a489470e5a-secret-metrics-server-tls\") pod \"metrics-server-74cbbd4d86-rjp57\" (UID: \"9e8864ce-4a84-4e3c-b838-a1a489470e5a\") " pod="openshift-monitoring/metrics-server-74cbbd4d86-rjp57" Jul 07 17:55:46.099998 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.099944 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:46.193352 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.193323 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e8864ce-4a84-4e3c-b838-a1a489470e5a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-74cbbd4d86-rjp57\" (UID: \"9e8864ce-4a84-4e3c-b838-a1a489470e5a\") " pod="openshift-monitoring/metrics-server-74cbbd4d86-rjp57" Jul 07 17:55:46.193516 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.193369 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9e8864ce-4a84-4e3c-b838-a1a489470e5a-audit-log\") pod \"metrics-server-74cbbd4d86-rjp57\" (UID: \"9e8864ce-4a84-4e3c-b838-a1a489470e5a\") " pod="openshift-monitoring/metrics-server-74cbbd4d86-rjp57" Jul 07 17:55:46.193591 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.193506 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9e8864ce-4a84-4e3c-b838-a1a489470e5a-client-ca-bundle\") pod \"metrics-server-74cbbd4d86-rjp57\" (UID: \"9e8864ce-4a84-4e3c-b838-a1a489470e5a\") " pod="openshift-monitoring/metrics-server-74cbbd4d86-rjp57" Jul 07 17:55:46.193591 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.193564 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9e8864ce-4a84-4e3c-b838-a1a489470e5a-secret-metrics-server-client-certs\") pod \"metrics-server-74cbbd4d86-rjp57\" (UID: \"9e8864ce-4a84-4e3c-b838-a1a489470e5a\") " pod="openshift-monitoring/metrics-server-74cbbd4d86-rjp57" Jul 07 17:55:46.193711 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.193601 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf8z6\" (UniqueName: \"kubernetes.io/projected/9e8864ce-4a84-4e3c-b838-a1a489470e5a-kube-api-access-cf8z6\") pod \"metrics-server-74cbbd4d86-rjp57\" (UID: \"9e8864ce-4a84-4e3c-b838-a1a489470e5a\") " pod="openshift-monitoring/metrics-server-74cbbd4d86-rjp57" Jul 07 17:55:46.193711 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.193655 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9e8864ce-4a84-4e3c-b838-a1a489470e5a-metrics-server-audit-profiles\") pod \"metrics-server-74cbbd4d86-rjp57\" (UID: \"9e8864ce-4a84-4e3c-b838-a1a489470e5a\") " pod="openshift-monitoring/metrics-server-74cbbd4d86-rjp57" Jul 07 17:55:46.193711 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.193689 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9e8864ce-4a84-4e3c-b838-a1a489470e5a-audit-log\") pod \"metrics-server-74cbbd4d86-rjp57\" (UID: \"9e8864ce-4a84-4e3c-b838-a1a489470e5a\") " pod="openshift-monitoring/metrics-server-74cbbd4d86-rjp57" Jul 07 17:55:46.193711 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.193690 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9e8864ce-4a84-4e3c-b838-a1a489470e5a-secret-metrics-server-tls\") pod \"metrics-server-74cbbd4d86-rjp57\" (UID: \"9e8864ce-4a84-4e3c-b838-a1a489470e5a\") " pod="openshift-monitoring/metrics-server-74cbbd4d86-rjp57" Jul 07 17:55:46.194207 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.194155 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e8864ce-4a84-4e3c-b838-a1a489470e5a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-74cbbd4d86-rjp57\" (UID: \"9e8864ce-4a84-4e3c-b838-a1a489470e5a\") " pod="openshift-monitoring/metrics-server-74cbbd4d86-rjp57" Jul 07 17:55:46.194968 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.194923 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9e8864ce-4a84-4e3c-b838-a1a489470e5a-metrics-server-audit-profiles\") pod \"metrics-server-74cbbd4d86-rjp57\" (UID: \"9e8864ce-4a84-4e3c-b838-a1a489470e5a\") " pod="openshift-monitoring/metrics-server-74cbbd4d86-rjp57" Jul 07 17:55:46.196747 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.196724 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9e8864ce-4a84-4e3c-b838-a1a489470e5a-secret-metrics-server-client-certs\") pod \"metrics-server-74cbbd4d86-rjp57\" (UID: \"9e8864ce-4a84-4e3c-b838-a1a489470e5a\") " pod="openshift-monitoring/metrics-server-74cbbd4d86-rjp57" Jul 07 17:55:46.196864 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.196824 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9e8864ce-4a84-4e3c-b838-a1a489470e5a-secret-metrics-server-tls\") pod \"metrics-server-74cbbd4d86-rjp57\" (UID: \"9e8864ce-4a84-4e3c-b838-a1a489470e5a\") " pod="openshift-monitoring/metrics-server-74cbbd4d86-rjp57" Jul 07 17:55:46.197231 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.197208 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9e8864ce-4a84-4e3c-b838-a1a489470e5a-client-ca-bundle\") pod \"metrics-server-74cbbd4d86-rjp57\" (UID: \"9e8864ce-4a84-4e3c-b838-a1a489470e5a\") " pod="openshift-monitoring/metrics-server-74cbbd4d86-rjp57" Jul 07 17:55:46.200712 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.200679 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:46.206958 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.206930 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf8z6\" (UniqueName: \"kubernetes.io/projected/9e8864ce-4a84-4e3c-b838-a1a489470e5a-kube-api-access-cf8z6\") pod \"metrics-server-74cbbd4d86-rjp57\" (UID: \"9e8864ce-4a84-4e3c-b838-a1a489470e5a\") " pod="openshift-monitoring/metrics-server-74cbbd4d86-rjp57" Jul 07 17:55:46.264320 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.264289 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-9f7gk"] Jul 07 17:55:46.268636 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.268612 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9f7gk" Jul 07 17:55:46.275163 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.275078 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-4pcmm\"" Jul 07 17:55:46.275163 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.275099 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 07 17:55:46.284432 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.284376 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-9f7gk"] Jul 07 17:55:46.295066 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.295034 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/40a8daa5-ff78-4901-8230-f4d32a6ffd27-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-9f7gk\" (UID: \"40a8daa5-ff78-4901-8230-f4d32a6ffd27\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9f7gk" Jul 07 17:55:46.295197 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.295091 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtm44\" (UniqueName: \"kubernetes.io/projected/40a8daa5-ff78-4901-8230-f4d32a6ffd27-kube-api-access-wtm44\") pod \"monitoring-plugin-d5dcff7cf-9f7gk\" (UID: \"40a8daa5-ff78-4901-8230-f4d32a6ffd27\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9f7gk" Jul 07 17:55:46.316206 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.316170 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:46.385234 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.385136 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-74cbbd4d86-rjp57" Jul 07 17:55:46.396500 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.396475 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/40a8daa5-ff78-4901-8230-f4d32a6ffd27-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-9f7gk\" (UID: \"40a8daa5-ff78-4901-8230-f4d32a6ffd27\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9f7gk" Jul 07 17:55:46.396584 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.396525 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtm44\" (UniqueName: \"kubernetes.io/projected/40a8daa5-ff78-4901-8230-f4d32a6ffd27-kube-api-access-wtm44\") pod \"monitoring-plugin-d5dcff7cf-9f7gk\" (UID: \"40a8daa5-ff78-4901-8230-f4d32a6ffd27\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9f7gk" Jul 07 17:55:46.399136 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.399107 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/40a8daa5-ff78-4901-8230-f4d32a6ffd27-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-9f7gk\" (UID: \"40a8daa5-ff78-4901-8230-f4d32a6ffd27\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9f7gk" Jul 07 17:55:46.408153 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.408129 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtm44\" (UniqueName: \"kubernetes.io/projected/40a8daa5-ff78-4901-8230-f4d32a6ffd27-kube-api-access-wtm44\") pod \"monitoring-plugin-d5dcff7cf-9f7gk\" (UID: \"40a8daa5-ff78-4901-8230-f4d32a6ffd27\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9f7gk" Jul 07 17:55:46.465217 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.465173 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:46.526855 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.526820 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-74cbbd4d86-rjp57"] Jul 07 17:55:46.529730 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:55:46.529698 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e8864ce_4a84_4e3c_b838_a1a489470e5a.slice/crio-ed699d0fc55a58239a99485b6704fb6c74d9c069b1decdbbb3ddb87773a325f6 WatchSource:0}: Error finding container ed699d0fc55a58239a99485b6704fb6c74d9c069b1decdbbb3ddb87773a325f6: Status 404 returned error can't find the container with id ed699d0fc55a58239a99485b6704fb6c74d9c069b1decdbbb3ddb87773a325f6 Jul 07 17:55:46.564890 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.564854 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:46.580188 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.580163 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9f7gk" Jul 07 17:55:46.725421 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.725388 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-9f7gk"] Jul 07 17:55:46.728270 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:55:46.728238 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40a8daa5_ff78_4901_8230_f4d32a6ffd27.slice/crio-d0ffbc3a72c22f14121ba004164b1cc8b9986f7b2d3f031693c54da0e9adfd4d WatchSource:0}: Error finding container d0ffbc3a72c22f14121ba004164b1cc8b9986f7b2d3f031693c54da0e9adfd4d: Status 404 returned error can't find the container with id d0ffbc3a72c22f14121ba004164b1cc8b9986f7b2d3f031693c54da0e9adfd4d Jul 07 17:55:46.979534 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.979439 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-74cbbd4d86-rjp57" event={"ID":"9e8864ce-4a84-4e3c-b838-a1a489470e5a","Type":"ContainerStarted","Data":"ed699d0fc55a58239a99485b6704fb6c74d9c069b1decdbbb3ddb87773a325f6"} Jul 07 17:55:46.981100 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.981073 2583 generic.go:356] "Generic (PLEG): container finished" podID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerID="ed946406f62803356e68651ef444ecfa020eb64bb4e1a4d936cf57ed9c3313e6" exitCode=0 Jul 07 17:55:46.981225 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.981110 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e","Type":"ContainerDied","Data":"ed946406f62803356e68651ef444ecfa020eb64bb4e1a4d936cf57ed9c3313e6"} Jul 07 17:55:46.983995 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.983968 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" event={"ID":"4a3bebfc-1f1e-4bb2-a02a-7174390c38b5","Type":"ContainerStarted","Data":"1851d1e14c48856efed9b316b6dd993a09d4be0c659004ef45b27c6dc3682282"} Jul 07 17:55:46.984095 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.984042 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" event={"ID":"4a3bebfc-1f1e-4bb2-a02a-7174390c38b5","Type":"ContainerStarted","Data":"3bea84a69a943a8c50e5a4407d6fc914225a54d89937f9a95acdfd242f58ee5a"} Jul 07 17:55:46.984095 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.984084 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" event={"ID":"4a3bebfc-1f1e-4bb2-a02a-7174390c38b5","Type":"ContainerStarted","Data":"90bc8c837e1c766f459931e6a6c3263c18d7d69910f6266c5925e692b30a63d5"} Jul 07 17:55:46.985206 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:46.985184 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9f7gk" event={"ID":"40a8daa5-ff78-4901-8230-f4d32a6ffd27","Type":"ContainerStarted","Data":"d0ffbc3a72c22f14121ba004164b1cc8b9986f7b2d3f031693c54da0e9adfd4d"} Jul 07 17:55:47.924750 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:47.924713 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 17:55:47.929384 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:47.929355 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:47.931433 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:47.931401 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:47.932191 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:47.932165 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 07 17:55:47.932689 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:47.932668 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 07 17:55:47.933862 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:47.933843 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 07 17:55:47.934348 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:47.934328 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 07 17:55:47.934484 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:47.934370 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5j0pnigmdrias\"" Jul 07 17:55:47.934577 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:47.934494 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 07 17:55:47.934713 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:47.934693 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 07 17:55:47.934836 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:47.934814 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 07 17:55:47.934958 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:47.934881 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 07 17:55:47.935035 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:47.934707 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 07 17:55:47.935035 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:47.934756 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 07 17:55:47.935035 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:47.935023 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 07 17:55:47.935182 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:47.935105 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 07 17:55:47.935182 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:47.935143 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 07 17:55:47.935807 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:47.935790 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6v2jb\"" Jul 07 17:55:47.942538 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:47.942513 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 07 17:55:47.950616 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:47.950590 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 17:55:47.997971 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:47.997934 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" event={"ID":"4a3bebfc-1f1e-4bb2-a02a-7174390c38b5","Type":"ContainerStarted","Data":"8ff42027dacaa5d4b1dac5e02a009df42c10e623645f1ee0496f27b0b22cc6e6"} Jul 07 17:55:47.998149 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:47.998008 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" event={"ID":"4a3bebfc-1f1e-4bb2-a02a-7174390c38b5","Type":"ContainerStarted","Data":"a6dd92d35be556772326ee809aacd6570c5093bd401dd2e58426a6b9fafd2d45"} Jul 07 17:55:47.998149 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:47.998041 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" event={"ID":"4a3bebfc-1f1e-4bb2-a02a-7174390c38b5","Type":"ContainerStarted","Data":"c2e315f3271ee2261aac7a82a2ea48bab493fafa59ce3e95850c743ab0172952"} Jul 07 17:55:47.998149 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:47.998135 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" Jul 07 17:55:48.014247 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.014191 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.014247 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.014234 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.014470 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.014277 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr9k6\" (UniqueName: \"kubernetes.io/projected/25545dec-41dd-4edf-9906-12bc2d5be5b2-kube-api-access-kr9k6\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.014470 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.014314 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.014470 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.014353 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.014470 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.014401 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/25545dec-41dd-4edf-9906-12bc2d5be5b2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.014627 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.014492 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.014627 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.014533 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.014627 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.014558 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25545dec-41dd-4edf-9906-12bc2d5be5b2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.014627 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.014587 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.014627 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.014604 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-web-config\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.014627 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.014626 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-config\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.014820 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.014640 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25545dec-41dd-4edf-9906-12bc2d5be5b2-config-out\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.014820 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.014671 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.014820 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.014696 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.014820 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.014716 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.014820 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.014734 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.014820 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.014754 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.014820 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.014769 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.014820 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.014790 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.025511 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.025454 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" podStartSLOduration=2.185786751 podStartE2EDuration="5.025436912s" totalImagesPullingTime="2.839650161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:55:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:55:48.025290218 +0000 UTC m=+97.107893923" watchObservedRunningTime="2026-07-07 17:55:48.025436912 +0000 UTC m=+97.108040616" Jul 07 17:55:48.115784 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.115739 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/25545dec-41dd-4edf-9906-12bc2d5be5b2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.115990 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.115808 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.115990 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.115842 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.115990 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.115884 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25545dec-41dd-4edf-9906-12bc2d5be5b2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.115990 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.115941 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.115990 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.115963 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-web-config\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.116256 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.115993 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-config\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.116256 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.116016 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25545dec-41dd-4edf-9906-12bc2d5be5b2-config-out\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.116256 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.116088 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.116256 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.116122 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.116256 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.116181 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.116256 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.116211 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.116256 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.116246 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.116577 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.116270 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.116577 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.116300 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.116577 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.116330 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.116577 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.116358 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.116577 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.116405 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr9k6\" (UniqueName: \"kubernetes.io/projected/25545dec-41dd-4edf-9906-12bc2d5be5b2-kube-api-access-kr9k6\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.116577 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.116433 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.116577 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.116497 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.117939 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.117642 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/25545dec-41dd-4edf-9906-12bc2d5be5b2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.118518 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.118327 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.119553 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.118642 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.119553 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.119445 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.120305 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.120285 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.120420 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.120396 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.120880 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.120856 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.122080 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.122055 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.122294 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.122266 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.123657 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.123632 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.124088 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.124069 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-web-config\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.124215 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.124189 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.124465 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.124443 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-config\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.125325 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.125065 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.125325 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.125318 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25545dec-41dd-4edf-9906-12bc2d5be5b2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.125570 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.125545 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25545dec-41dd-4edf-9906-12bc2d5be5b2-config-out\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.126124 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.126101 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.126600 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.126575 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.128545 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.128504 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.130662 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.130617 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr9k6\" (UniqueName: \"kubernetes.io/projected/25545dec-41dd-4edf-9906-12bc2d5be5b2-kube-api-access-kr9k6\") pod \"prometheus-k8s-0\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.248746 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.248662 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:48.423113 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.423085 2583 scope.go:117] "RemoveContainer" containerID="e6b1245a376909f6b71c28cdd2f927c690e644ffe4e5422b0b5b82ad4136de58" Jul 07 17:55:48.962421 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:48.962317 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 17:55:48.967257 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:55:48.967181 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25545dec_41dd_4edf_9906_12bc2d5be5b2.slice/crio-e03c997ec421a0b686e40e8da32d0e87b9b1aa5402a5799bdb10a38b0e698b5a WatchSource:0}: Error finding container e03c997ec421a0b686e40e8da32d0e87b9b1aa5402a5799bdb10a38b0e698b5a: Status 404 returned error can't find the container with id e03c997ec421a0b686e40e8da32d0e87b9b1aa5402a5799bdb10a38b0e698b5a Jul 07 17:55:49.007454 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:49.006831 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9f7gk" event={"ID":"40a8daa5-ff78-4901-8230-f4d32a6ffd27","Type":"ContainerStarted","Data":"1f68b0266096573d4f493244bd5606b354265b222cea534d2991868909e53555"} Jul 07 17:55:49.007454 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:49.007357 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9f7gk" Jul 07 17:55:49.010945 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:49.010773 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-74cbbd4d86-rjp57" event={"ID":"9e8864ce-4a84-4e3c-b838-a1a489470e5a","Type":"ContainerStarted","Data":"efb5ad68e738428c888d1bc7ac3bfba915e6f593974451f96cc52ad42efa7499"} Jul 07 17:55:49.013554 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:49.013527 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wqfm4_997df85c-7632-47b5-a8a6-1c37f3346251/console-operator/2.log" Jul 07 17:55:49.013686 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:49.013660 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" event={"ID":"997df85c-7632-47b5-a8a6-1c37f3346251","Type":"ContainerStarted","Data":"51824750b2bb4c67169657e29cfa391d83d0b953b187abf58f99a1b0789869bf"} Jul 07 17:55:49.014133 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:49.013935 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" Jul 07 17:55:49.014893 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:49.014450 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9f7gk" Jul 07 17:55:49.017157 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:49.017129 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e","Type":"ContainerStarted","Data":"454631df6a9b2b3295e79287a9bfdd4b5779195d00959bb24d8a198f5b952658"} Jul 07 17:55:49.017255 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:49.017201 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e","Type":"ContainerStarted","Data":"f5f91bb70d74d203434c63cadc633bb04fc16be5ac8fd4f65784d3780342ca5c"} Jul 07 17:55:49.018736 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:49.018710 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25545dec-41dd-4edf-9906-12bc2d5be5b2","Type":"ContainerStarted","Data":"e03c997ec421a0b686e40e8da32d0e87b9b1aa5402a5799bdb10a38b0e698b5a"} Jul 07 17:55:49.048824 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:49.048770 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" podStartSLOduration=56.400020806 podStartE2EDuration="59.048745619s" totalImagesPullingTime="2.648724813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:54:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:55:49.046983671 +0000 UTC m=+98.129587376" watchObservedRunningTime="2026-07-07 17:55:49.048745619 +0000 UTC m=+98.131349324" Jul 07 17:55:49.058244 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:49.051281 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-9f7gk" podStartSLOduration=1.001129822 podStartE2EDuration="3.051260313s" totalImagesPullingTime="2.050130491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:55:49.02843839 +0000 UTC m=+98.111042097" watchObservedRunningTime="2026-07-07 17:55:49.051260313 +0000 UTC m=+98.133864018" Jul 07 17:55:49.089275 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:49.089218 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-74cbbd4d86-rjp57" podStartSLOduration=0.844061981 podStartE2EDuration="3.08919632s" totalImagesPullingTime="2.245134339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 17:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:55:49.087740856 +0000 UTC m=+98.170344557" watchObservedRunningTime="2026-07-07 17:55:49.08919632 +0000 UTC m=+98.171800026" Jul 07 17:55:49.484439 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:49.484357 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-wqfm4" Jul 07 17:55:49.793366 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:49.793336 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:49.878879 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:49.878848 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:49.982944 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:49.982881 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:50.026358 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:50.026323 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e","Type":"ContainerStarted","Data":"d7c7ff4f026cb365e2769577c4c3f51373c0beaa0b4f6f77639b4a6019ce6f35"} Jul 07 17:55:50.026518 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:50.026405 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e","Type":"ContainerStarted","Data":"1b836662ca0d207006de2191888c0098f5bdd3b7b98ad37ee628b3dedb24ae1d"} Jul 07 17:55:50.026518 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:50.026444 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e","Type":"ContainerStarted","Data":"4cd2e37965ad217e2f92e2d5a0988791bc6b14b5de65810376d8b2034f2d08c7"} Jul 07 17:55:50.026518 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:50.026479 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e","Type":"ContainerStarted","Data":"0449587351ebfa9b71be33144073ce8e78b08108f88949f3e10fc6eb409b96cb"} Jul 07 17:55:50.027817 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:50.027793 2583 generic.go:356] "Generic (PLEG): container finished" podID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerID="d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1" exitCode=0 Jul 07 17:55:50.027962 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:50.027888 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25545dec-41dd-4edf-9906-12bc2d5be5b2","Type":"ContainerDied","Data":"d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1"} Jul 07 17:55:50.063618 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:50.063457 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.980793593 podStartE2EDuration="8.063432605s" totalImagesPullingTime="3.061878565s" totalInitContainerRuntime="20.760447ms" isStatefulPod=false podCreationTimestamp="2026-07-07 17:55:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:55:50.062529044 +0000 UTC m=+99.145132749" watchObservedRunningTime="2026-07-07 17:55:50.063432605 +0000 UTC m=+99.146036312" Jul 07 17:55:50.352772 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:50.352669 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:50.460110 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:50.460067 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:50.769970 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:50.769860 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:51.387847 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:51.387803 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:51.546264 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:51.546211 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:51.645275 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:51.645187 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:51.795354 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:51.795319 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:52.099531 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:52.099487 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:52.173344 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:52.173307 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:54.027173 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:54.027141 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-568b77c849-rc4mb" Jul 07 17:55:54.048918 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:54.048869 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25545dec-41dd-4edf-9906-12bc2d5be5b2","Type":"ContainerStarted","Data":"5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53"} Jul 07 17:55:54.049033 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:54.048965 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25545dec-41dd-4edf-9906-12bc2d5be5b2","Type":"ContainerStarted","Data":"4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b"} Jul 07 17:55:54.049033 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:54.048994 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25545dec-41dd-4edf-9906-12bc2d5be5b2","Type":"ContainerStarted","Data":"7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d"} Jul 07 17:55:54.175863 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:54.175835 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:54.272576 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:54.272547 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:54.555991 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:54.555955 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:54.659613 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:54.659580 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:54.867377 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:54.867303 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bmdgm" Jul 07 17:55:55.056240 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:55.056205 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25545dec-41dd-4edf-9906-12bc2d5be5b2","Type":"ContainerStarted","Data":"053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd"} Jul 07 17:55:55.056716 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:55.056269 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25545dec-41dd-4edf-9906-12bc2d5be5b2","Type":"ContainerStarted","Data":"44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b"} Jul 07 17:55:55.056716 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:55.056296 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25545dec-41dd-4edf-9906-12bc2d5be5b2","Type":"ContainerStarted","Data":"d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf"} Jul 07 17:55:55.089865 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:55.089812 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.502238253 podStartE2EDuration="8.089791054s" totalImagesPullingTime="3.556000944s" totalInitContainerRuntime="31.551857ms" isStatefulPod=false podCreationTimestamp="2026-07-07 17:55:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:55:55.087148812 +0000 UTC m=+104.169752523" watchObservedRunningTime="2026-07-07 17:55:55.089791054 +0000 UTC m=+104.172394757" Jul 07 17:55:58.249211 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:58.249171 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:55:59.665054 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:59.665012 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:55:59.788995 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:55:59.788953 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:01.424056 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:01.424019 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:01.538530 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:01.538488 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:02.002469 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:02.002428 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:02.109603 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:02.109562 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:02.203257 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:02.203219 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:03.678705 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:03.678664 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:03.788952 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:03.788888 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:04.629009 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:04.628974 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:04.720342 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:04.720309 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:06.386035 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:06.385996 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-74cbbd4d86-rjp57" Jul 07 17:56:06.386514 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:06.386091 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-74cbbd4d86-rjp57" Jul 07 17:56:09.271278 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:09.271245 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:09.365454 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:09.365417 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:10.968135 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:10.968102 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:11.069428 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:11.069399 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:11.173259 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:11.173224 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:13.314126 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:13.314096 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:13.384407 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:13.384377 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:17.312447 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:17.312411 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:17.447596 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:17.447553 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:22.147071 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:22.147014 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:22.170823 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:22.170794 2583 generic.go:356] "Generic (PLEG): container finished" podID="2ed38cd7-9e63-4b1b-97cb-86748bb60a04" containerID="33ceb2eccac825f5a8096456f9c15020273f589f72565d3ac078383406a3f29e" exitCode=0 Jul 07 17:56:22.170998 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:22.170869 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bjv6p" event={"ID":"2ed38cd7-9e63-4b1b-97cb-86748bb60a04","Type":"ContainerDied","Data":"33ceb2eccac825f5a8096456f9c15020273f589f72565d3ac078383406a3f29e"} Jul 07 17:56:22.171261 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:22.171246 2583 scope.go:117] "RemoveContainer" containerID="33ceb2eccac825f5a8096456f9c15020273f589f72565d3ac078383406a3f29e" Jul 07 17:56:22.312720 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:22.312682 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:23.177170 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:23.177133 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bjv6p" event={"ID":"2ed38cd7-9e63-4b1b-97cb-86748bb60a04","Type":"ContainerStarted","Data":"d94a47d58db4b995a165708f814f65c445728ada0b82a38fecc7074f9f227a83"} Jul 07 17:56:23.470950 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:23.470807 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:23.572608 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:23.572568 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:24.470249 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:24.470218 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:26.390877 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:26.390838 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-74cbbd4d86-rjp57" Jul 07 17:56:26.394835 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:26.394814 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-74cbbd4d86-rjp57" Jul 07 17:56:32.172997 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:32.172959 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:32.309464 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:32.309424 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:56:48.249818 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:48.249774 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:56:48.266698 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:48.266673 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:56:48.311357 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:56:48.311330 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:00.987453 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:00.987414 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:57:01.088072 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:01.088040 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:57:02.013180 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:02.013088 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 17:57:02.013625 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:02.013460 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerName="alertmanager" containerID="cri-o://f5f91bb70d74d203434c63cadc633bb04fc16be5ac8fd4f65784d3780342ca5c" gracePeriod=120 Jul 07 17:57:02.013625 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:02.013527 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerName="kube-rbac-proxy-web" containerID="cri-o://0449587351ebfa9b71be33144073ce8e78b08108f88949f3e10fc6eb409b96cb" gracePeriod=120 Jul 07 17:57:02.013625 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:02.013528 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerName="kube-rbac-proxy-metric" containerID="cri-o://1b836662ca0d207006de2191888c0098f5bdd3b7b98ad37ee628b3dedb24ae1d" gracePeriod=120 Jul 07 17:57:02.013779 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:02.013589 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerName="kube-rbac-proxy" containerID="cri-o://4cd2e37965ad217e2f92e2d5a0988791bc6b14b5de65810376d8b2034f2d08c7" gracePeriod=120 Jul 07 17:57:02.013779 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:02.013559 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerName="config-reloader" containerID="cri-o://454631df6a9b2b3295e79287a9bfdd4b5779195d00959bb24d8a198f5b952658" gracePeriod=120 Jul 07 17:57:02.013779 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:02.013683 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerName="prom-label-proxy" containerID="cri-o://d7c7ff4f026cb365e2769577c4c3f51373c0beaa0b4f6f77639b4a6019ce6f35" gracePeriod=120 Jul 07 17:57:02.354730 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:02.354685 2583 generic.go:356] "Generic (PLEG): container finished" podID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerID="d7c7ff4f026cb365e2769577c4c3f51373c0beaa0b4f6f77639b4a6019ce6f35" exitCode=0 Jul 07 17:57:02.354730 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:02.354722 2583 generic.go:356] "Generic (PLEG): container finished" podID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerID="4cd2e37965ad217e2f92e2d5a0988791bc6b14b5de65810376d8b2034f2d08c7" exitCode=0 Jul 07 17:57:02.354730 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:02.354729 2583 generic.go:356] "Generic (PLEG): container finished" podID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerID="454631df6a9b2b3295e79287a9bfdd4b5779195d00959bb24d8a198f5b952658" exitCode=0 Jul 07 17:57:02.354730 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:02.354734 2583 generic.go:356] "Generic (PLEG): container finished" podID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerID="f5f91bb70d74d203434c63cadc633bb04fc16be5ac8fd4f65784d3780342ca5c" exitCode=0 Jul 07 17:57:02.355011 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:02.354755 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e","Type":"ContainerDied","Data":"d7c7ff4f026cb365e2769577c4c3f51373c0beaa0b4f6f77639b4a6019ce6f35"} Jul 07 17:57:02.355011 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:02.354827 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e","Type":"ContainerDied","Data":"4cd2e37965ad217e2f92e2d5a0988791bc6b14b5de65810376d8b2034f2d08c7"} Jul 07 17:57:02.355011 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:02.354853 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e","Type":"ContainerDied","Data":"454631df6a9b2b3295e79287a9bfdd4b5779195d00959bb24d8a198f5b952658"} Jul 07 17:57:02.355011 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:02.354883 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e","Type":"ContainerDied","Data":"f5f91bb70d74d203434c63cadc633bb04fc16be5ac8fd4f65784d3780342ca5c"} Jul 07 17:57:03.263215 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.263188 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.304684 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.304656 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-alertmanager-main-db\") pod \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " Jul 07 17:57:03.304876 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.304736 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-cluster-tls-config\") pod \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " Jul 07 17:57:03.304876 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.304763 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-metrics-client-ca\") pod \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " Jul 07 17:57:03.304876 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.304806 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-tls-assets\") pod \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " Jul 07 17:57:03.304876 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.304828 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-config-volume\") pod \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " Jul 07 17:57:03.304876 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.304855 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-config-out\") pod \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " Jul 07 17:57:03.305158 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.304932 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-secret-alertmanager-main-tls\") pod \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " Jul 07 17:57:03.305158 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.304963 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-web-config\") pod \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " Jul 07 17:57:03.305158 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.305014 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " Jul 07 17:57:03.305158 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.305030 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" (UID: "d0a24e05-c4bb-4beb-b46e-ec0e58d0832e"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:57:03.305158 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.305045 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-secret-alertmanager-kube-rbac-proxy-web\") pod \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " Jul 07 17:57:03.305158 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.305105 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-secret-alertmanager-kube-rbac-proxy\") pod \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " Jul 07 17:57:03.305158 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.305143 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-alertmanager-trusted-ca-bundle\") pod \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " Jul 07 17:57:03.305510 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.305177 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wv2xr\" (UniqueName: \"kubernetes.io/projected/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-kube-api-access-wv2xr\") pod \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\" (UID: \"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e\") " Jul 07 17:57:03.305510 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.305478 2583 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-alertmanager-main-db\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:03.306508 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.306234 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" (UID: "d0a24e05-c4bb-4beb-b46e-ec0e58d0832e"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:57:03.306942 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.306760 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" (UID: "d0a24e05-c4bb-4beb-b46e-ec0e58d0832e"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:57:03.308619 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.308583 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" (UID: "d0a24e05-c4bb-4beb-b46e-ec0e58d0832e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:57:03.312365 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.312238 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" (UID: "d0a24e05-c4bb-4beb-b46e-ec0e58d0832e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:57:03.312365 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.312316 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-config-out" (OuterVolumeSpecName: "config-out") pod "d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" (UID: "d0a24e05-c4bb-4beb-b46e-ec0e58d0832e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:57:03.312730 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.312692 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-config-volume" (OuterVolumeSpecName: "config-volume") pod "d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" (UID: "d0a24e05-c4bb-4beb-b46e-ec0e58d0832e"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:57:03.313003 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.312979 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" (UID: "d0a24e05-c4bb-4beb-b46e-ec0e58d0832e"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:57:03.314249 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.314222 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-kube-api-access-wv2xr" (OuterVolumeSpecName: "kube-api-access-wv2xr") pod "d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" (UID: "d0a24e05-c4bb-4beb-b46e-ec0e58d0832e"). InnerVolumeSpecName "kube-api-access-wv2xr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:57:03.314734 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.314706 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" (UID: "d0a24e05-c4bb-4beb-b46e-ec0e58d0832e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:57:03.316705 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.316667 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" (UID: "d0a24e05-c4bb-4beb-b46e-ec0e58d0832e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:57:03.317832 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.317808 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" (UID: "d0a24e05-c4bb-4beb-b46e-ec0e58d0832e"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:57:03.325969 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.325879 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-web-config" (OuterVolumeSpecName: "web-config") pod "d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" (UID: "d0a24e05-c4bb-4beb-b46e-ec0e58d0832e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:57:03.362246 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.362214 2583 generic.go:356] "Generic (PLEG): container finished" podID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerID="1b836662ca0d207006de2191888c0098f5bdd3b7b98ad37ee628b3dedb24ae1d" exitCode=0 Jul 07 17:57:03.362246 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.362242 2583 generic.go:356] "Generic (PLEG): container finished" podID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerID="0449587351ebfa9b71be33144073ce8e78b08108f88949f3e10fc6eb409b96cb" exitCode=0 Jul 07 17:57:03.362485 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.362294 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.362485 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.362294 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e","Type":"ContainerDied","Data":"1b836662ca0d207006de2191888c0098f5bdd3b7b98ad37ee628b3dedb24ae1d"} Jul 07 17:57:03.362485 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.362419 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e","Type":"ContainerDied","Data":"0449587351ebfa9b71be33144073ce8e78b08108f88949f3e10fc6eb409b96cb"} Jul 07 17:57:03.362485 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.362467 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d0a24e05-c4bb-4beb-b46e-ec0e58d0832e","Type":"ContainerDied","Data":"862a59350aa9931a6a07b8ff33b2c32b243a61567dc6ab1e1e041ef4ff9a2d7f"} Jul 07 17:57:03.362485 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.362482 2583 scope.go:117] "RemoveContainer" containerID="d7c7ff4f026cb365e2769577c4c3f51373c0beaa0b4f6f77639b4a6019ce6f35" Jul 07 17:57:03.370741 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.370724 2583 scope.go:117] "RemoveContainer" containerID="1b836662ca0d207006de2191888c0098f5bdd3b7b98ad37ee628b3dedb24ae1d" Jul 07 17:57:03.379054 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.379033 2583 scope.go:117] "RemoveContainer" containerID="4cd2e37965ad217e2f92e2d5a0988791bc6b14b5de65810376d8b2034f2d08c7" Jul 07 17:57:03.387457 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.387432 2583 scope.go:117] "RemoveContainer" containerID="0449587351ebfa9b71be33144073ce8e78b08108f88949f3e10fc6eb409b96cb" Jul 07 17:57:03.389090 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.389069 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 17:57:03.394255 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.394231 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 17:57:03.396569 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.396553 2583 scope.go:117] "RemoveContainer" containerID="454631df6a9b2b3295e79287a9bfdd4b5779195d00959bb24d8a198f5b952658" Jul 07 17:57:03.404329 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.404312 2583 scope.go:117] "RemoveContainer" containerID="f5f91bb70d74d203434c63cadc633bb04fc16be5ac8fd4f65784d3780342ca5c" Jul 07 17:57:03.406642 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.406617 2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:03.406642 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.406640 2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:03.406775 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.406651 2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:03.406775 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.406662 2583 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:03.406775 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.406673 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv2xr\" (UniqueName: \"kubernetes.io/projected/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-kube-api-access-wv2xr\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:03.406775 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.406681 2583 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-cluster-tls-config\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:03.406775 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.406689 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-metrics-client-ca\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:03.406775 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.406698 2583 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-tls-assets\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:03.406775 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.406706 2583 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-config-volume\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:03.406775 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.406715 2583 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-config-out\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:03.406775 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.406722 2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-secret-alertmanager-main-tls\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:03.406775 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.406730 2583 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e-web-config\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:03.412281 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.412263 2583 scope.go:117] "RemoveContainer" containerID="ed946406f62803356e68651ef444ecfa020eb64bb4e1a4d936cf57ed9c3313e6" Jul 07 17:57:03.420617 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.420589 2583 scope.go:117] "RemoveContainer" containerID="d7c7ff4f026cb365e2769577c4c3f51373c0beaa0b4f6f77639b4a6019ce6f35" Jul 07 17:57:03.420894 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:57:03.420872 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7c7ff4f026cb365e2769577c4c3f51373c0beaa0b4f6f77639b4a6019ce6f35\": container with ID starting with d7c7ff4f026cb365e2769577c4c3f51373c0beaa0b4f6f77639b4a6019ce6f35 not found: ID does not exist" containerID="d7c7ff4f026cb365e2769577c4c3f51373c0beaa0b4f6f77639b4a6019ce6f35" Jul 07 17:57:03.420986 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.420917 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7c7ff4f026cb365e2769577c4c3f51373c0beaa0b4f6f77639b4a6019ce6f35"} err="failed to get container status \"d7c7ff4f026cb365e2769577c4c3f51373c0beaa0b4f6f77639b4a6019ce6f35\": rpc error: code = NotFound desc = could not find container \"d7c7ff4f026cb365e2769577c4c3f51373c0beaa0b4f6f77639b4a6019ce6f35\": container with ID starting with d7c7ff4f026cb365e2769577c4c3f51373c0beaa0b4f6f77639b4a6019ce6f35 not found: ID does not exist" Jul 07 17:57:03.420986 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.420953 2583 scope.go:117] "RemoveContainer" containerID="1b836662ca0d207006de2191888c0098f5bdd3b7b98ad37ee628b3dedb24ae1d" Jul 07 17:57:03.421247 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:57:03.421210 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b836662ca0d207006de2191888c0098f5bdd3b7b98ad37ee628b3dedb24ae1d\": container with ID starting with 1b836662ca0d207006de2191888c0098f5bdd3b7b98ad37ee628b3dedb24ae1d not found: ID does not exist" containerID="1b836662ca0d207006de2191888c0098f5bdd3b7b98ad37ee628b3dedb24ae1d" Jul 07 17:57:03.421356 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.421245 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b836662ca0d207006de2191888c0098f5bdd3b7b98ad37ee628b3dedb24ae1d"} err="failed to get container status \"1b836662ca0d207006de2191888c0098f5bdd3b7b98ad37ee628b3dedb24ae1d\": rpc error: code = NotFound desc = could not find container \"1b836662ca0d207006de2191888c0098f5bdd3b7b98ad37ee628b3dedb24ae1d\": container with ID starting with 1b836662ca0d207006de2191888c0098f5bdd3b7b98ad37ee628b3dedb24ae1d not found: ID does not exist" Jul 07 17:57:03.421356 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.421269 2583 scope.go:117] "RemoveContainer" containerID="4cd2e37965ad217e2f92e2d5a0988791bc6b14b5de65810376d8b2034f2d08c7" Jul 07 17:57:03.421618 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:57:03.421556 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cd2e37965ad217e2f92e2d5a0988791bc6b14b5de65810376d8b2034f2d08c7\": container with ID starting with 4cd2e37965ad217e2f92e2d5a0988791bc6b14b5de65810376d8b2034f2d08c7 not found: ID does not exist" containerID="4cd2e37965ad217e2f92e2d5a0988791bc6b14b5de65810376d8b2034f2d08c7" Jul 07 17:57:03.421782 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.421758 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cd2e37965ad217e2f92e2d5a0988791bc6b14b5de65810376d8b2034f2d08c7"} err="failed to get container status \"4cd2e37965ad217e2f92e2d5a0988791bc6b14b5de65810376d8b2034f2d08c7\": rpc error: code = NotFound desc = could not find container \"4cd2e37965ad217e2f92e2d5a0988791bc6b14b5de65810376d8b2034f2d08c7\": container with ID starting with 4cd2e37965ad217e2f92e2d5a0988791bc6b14b5de65810376d8b2034f2d08c7 not found: ID does not exist" Jul 07 17:57:03.421889 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.421787 2583 scope.go:117] "RemoveContainer" containerID="0449587351ebfa9b71be33144073ce8e78b08108f88949f3e10fc6eb409b96cb" Jul 07 17:57:03.422278 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:57:03.422252 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0449587351ebfa9b71be33144073ce8e78b08108f88949f3e10fc6eb409b96cb\": container with ID starting with 0449587351ebfa9b71be33144073ce8e78b08108f88949f3e10fc6eb409b96cb not found: ID does not exist" containerID="0449587351ebfa9b71be33144073ce8e78b08108f88949f3e10fc6eb409b96cb" Jul 07 17:57:03.422344 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.422283 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0449587351ebfa9b71be33144073ce8e78b08108f88949f3e10fc6eb409b96cb"} err="failed to get container status \"0449587351ebfa9b71be33144073ce8e78b08108f88949f3e10fc6eb409b96cb\": rpc error: code = NotFound desc = could not find container \"0449587351ebfa9b71be33144073ce8e78b08108f88949f3e10fc6eb409b96cb\": container with ID starting with 0449587351ebfa9b71be33144073ce8e78b08108f88949f3e10fc6eb409b96cb not found: ID does not exist" Jul 07 17:57:03.422344 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.422312 2583 scope.go:117] "RemoveContainer" containerID="454631df6a9b2b3295e79287a9bfdd4b5779195d00959bb24d8a198f5b952658" Jul 07 17:57:03.422606 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:57:03.422582 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"454631df6a9b2b3295e79287a9bfdd4b5779195d00959bb24d8a198f5b952658\": container with ID starting with 454631df6a9b2b3295e79287a9bfdd4b5779195d00959bb24d8a198f5b952658 not found: ID does not exist" containerID="454631df6a9b2b3295e79287a9bfdd4b5779195d00959bb24d8a198f5b952658" Jul 07 17:57:03.422711 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.422619 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454631df6a9b2b3295e79287a9bfdd4b5779195d00959bb24d8a198f5b952658"} err="failed to get container status \"454631df6a9b2b3295e79287a9bfdd4b5779195d00959bb24d8a198f5b952658\": rpc error: code = NotFound desc = could not find container \"454631df6a9b2b3295e79287a9bfdd4b5779195d00959bb24d8a198f5b952658\": container with ID starting with 454631df6a9b2b3295e79287a9bfdd4b5779195d00959bb24d8a198f5b952658 not found: ID does not exist" Jul 07 17:57:03.422711 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.422640 2583 scope.go:117] "RemoveContainer" containerID="f5f91bb70d74d203434c63cadc633bb04fc16be5ac8fd4f65784d3780342ca5c" Jul 07 17:57:03.423030 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:57:03.422985 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5f91bb70d74d203434c63cadc633bb04fc16be5ac8fd4f65784d3780342ca5c\": container with ID starting with f5f91bb70d74d203434c63cadc633bb04fc16be5ac8fd4f65784d3780342ca5c not found: ID does not exist" containerID="f5f91bb70d74d203434c63cadc633bb04fc16be5ac8fd4f65784d3780342ca5c" Jul 07 17:57:03.423137 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.423037 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5f91bb70d74d203434c63cadc633bb04fc16be5ac8fd4f65784d3780342ca5c"} err="failed to get container status \"f5f91bb70d74d203434c63cadc633bb04fc16be5ac8fd4f65784d3780342ca5c\": rpc error: code = NotFound desc = could not find container \"f5f91bb70d74d203434c63cadc633bb04fc16be5ac8fd4f65784d3780342ca5c\": container with ID starting with f5f91bb70d74d203434c63cadc633bb04fc16be5ac8fd4f65784d3780342ca5c not found: ID does not exist" Jul 07 17:57:03.423137 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.423059 2583 scope.go:117] "RemoveContainer" containerID="ed946406f62803356e68651ef444ecfa020eb64bb4e1a4d936cf57ed9c3313e6" Jul 07 17:57:03.423400 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:57:03.423373 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed946406f62803356e68651ef444ecfa020eb64bb4e1a4d936cf57ed9c3313e6\": container with ID starting with ed946406f62803356e68651ef444ecfa020eb64bb4e1a4d936cf57ed9c3313e6 not found: ID does not exist" containerID="ed946406f62803356e68651ef444ecfa020eb64bb4e1a4d936cf57ed9c3313e6" Jul 07 17:57:03.423480 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.423410 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed946406f62803356e68651ef444ecfa020eb64bb4e1a4d936cf57ed9c3313e6"} err="failed to get container status \"ed946406f62803356e68651ef444ecfa020eb64bb4e1a4d936cf57ed9c3313e6\": rpc error: code = NotFound desc = could not find container \"ed946406f62803356e68651ef444ecfa020eb64bb4e1a4d936cf57ed9c3313e6\": container with ID starting with ed946406f62803356e68651ef444ecfa020eb64bb4e1a4d936cf57ed9c3313e6 not found: ID does not exist" Jul 07 17:57:03.423480 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.423428 2583 scope.go:117] "RemoveContainer" containerID="d7c7ff4f026cb365e2769577c4c3f51373c0beaa0b4f6f77639b4a6019ce6f35" Jul 07 17:57:03.423692 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.423669 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7c7ff4f026cb365e2769577c4c3f51373c0beaa0b4f6f77639b4a6019ce6f35"} err="failed to get container status \"d7c7ff4f026cb365e2769577c4c3f51373c0beaa0b4f6f77639b4a6019ce6f35\": rpc error: code = NotFound desc = could not find container \"d7c7ff4f026cb365e2769577c4c3f51373c0beaa0b4f6f77639b4a6019ce6f35\": container with ID starting with d7c7ff4f026cb365e2769577c4c3f51373c0beaa0b4f6f77639b4a6019ce6f35 not found: ID does not exist" Jul 07 17:57:03.423751 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.423695 2583 scope.go:117] "RemoveContainer" containerID="1b836662ca0d207006de2191888c0098f5bdd3b7b98ad37ee628b3dedb24ae1d" Jul 07 17:57:03.423972 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.423951 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b836662ca0d207006de2191888c0098f5bdd3b7b98ad37ee628b3dedb24ae1d"} err="failed to get container status \"1b836662ca0d207006de2191888c0098f5bdd3b7b98ad37ee628b3dedb24ae1d\": rpc error: code = NotFound desc = could not find container \"1b836662ca0d207006de2191888c0098f5bdd3b7b98ad37ee628b3dedb24ae1d\": container with ID starting with 1b836662ca0d207006de2191888c0098f5bdd3b7b98ad37ee628b3dedb24ae1d not found: ID does not exist" Jul 07 17:57:03.424061 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.423973 2583 scope.go:117] "RemoveContainer" containerID="4cd2e37965ad217e2f92e2d5a0988791bc6b14b5de65810376d8b2034f2d08c7" Jul 07 17:57:03.424238 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.424219 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cd2e37965ad217e2f92e2d5a0988791bc6b14b5de65810376d8b2034f2d08c7"} err="failed to get container status \"4cd2e37965ad217e2f92e2d5a0988791bc6b14b5de65810376d8b2034f2d08c7\": rpc error: code = NotFound desc = could not find container \"4cd2e37965ad217e2f92e2d5a0988791bc6b14b5de65810376d8b2034f2d08c7\": container with ID starting with 4cd2e37965ad217e2f92e2d5a0988791bc6b14b5de65810376d8b2034f2d08c7 not found: ID does not exist" Jul 07 17:57:03.424304 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.424239 2583 scope.go:117] "RemoveContainer" containerID="0449587351ebfa9b71be33144073ce8e78b08108f88949f3e10fc6eb409b96cb" Jul 07 17:57:03.424499 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.424480 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0449587351ebfa9b71be33144073ce8e78b08108f88949f3e10fc6eb409b96cb"} err="failed to get container status \"0449587351ebfa9b71be33144073ce8e78b08108f88949f3e10fc6eb409b96cb\": rpc error: code = NotFound desc = could not find container \"0449587351ebfa9b71be33144073ce8e78b08108f88949f3e10fc6eb409b96cb\": container with ID starting with 0449587351ebfa9b71be33144073ce8e78b08108f88949f3e10fc6eb409b96cb not found: ID does not exist" Jul 07 17:57:03.424562 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.424500 2583 scope.go:117] "RemoveContainer" containerID="454631df6a9b2b3295e79287a9bfdd4b5779195d00959bb24d8a198f5b952658" Jul 07 17:57:03.424724 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.424706 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454631df6a9b2b3295e79287a9bfdd4b5779195d00959bb24d8a198f5b952658"} err="failed to get container status \"454631df6a9b2b3295e79287a9bfdd4b5779195d00959bb24d8a198f5b952658\": rpc error: code = NotFound desc = could not find container \"454631df6a9b2b3295e79287a9bfdd4b5779195d00959bb24d8a198f5b952658\": container with ID starting with 454631df6a9b2b3295e79287a9bfdd4b5779195d00959bb24d8a198f5b952658 not found: ID does not exist" Jul 07 17:57:03.424787 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.424724 2583 scope.go:117] "RemoveContainer" containerID="f5f91bb70d74d203434c63cadc633bb04fc16be5ac8fd4f65784d3780342ca5c" Jul 07 17:57:03.424948 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.424926 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5f91bb70d74d203434c63cadc633bb04fc16be5ac8fd4f65784d3780342ca5c"} err="failed to get container status \"f5f91bb70d74d203434c63cadc633bb04fc16be5ac8fd4f65784d3780342ca5c\": rpc error: code = NotFound desc = could not find container \"f5f91bb70d74d203434c63cadc633bb04fc16be5ac8fd4f65784d3780342ca5c\": container with ID starting with f5f91bb70d74d203434c63cadc633bb04fc16be5ac8fd4f65784d3780342ca5c not found: ID does not exist" Jul 07 17:57:03.424948 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.424948 2583 scope.go:117] "RemoveContainer" containerID="ed946406f62803356e68651ef444ecfa020eb64bb4e1a4d936cf57ed9c3313e6" Jul 07 17:57:03.425184 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.425168 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed946406f62803356e68651ef444ecfa020eb64bb4e1a4d936cf57ed9c3313e6"} err="failed to get container status \"ed946406f62803356e68651ef444ecfa020eb64bb4e1a4d936cf57ed9c3313e6\": rpc error: code = NotFound desc = could not find container \"ed946406f62803356e68651ef444ecfa020eb64bb4e1a4d936cf57ed9c3313e6\": container with ID starting with ed946406f62803356e68651ef444ecfa020eb64bb4e1a4d936cf57ed9c3313e6 not found: ID does not exist" Jul 07 17:57:03.428461 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.428441 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" path="/var/lib/kubelet/pods/d0a24e05-c4bb-4beb-b46e-ec0e58d0832e/volumes" Jul 07 17:57:03.428966 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.428952 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 17:57:03.429243 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.429232 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerName="prom-label-proxy" Jul 07 17:57:03.429281 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.429245 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerName="prom-label-proxy" Jul 07 17:57:03.429281 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.429254 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerName="kube-rbac-proxy-metric" Jul 07 17:57:03.429281 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.429260 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerName="kube-rbac-proxy-metric" Jul 07 17:57:03.429281 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.429273 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerName="kube-rbac-proxy-web" Jul 07 17:57:03.429281 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.429278 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerName="kube-rbac-proxy-web" Jul 07 17:57:03.429422 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.429286 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerName="config-reloader" Jul 07 17:57:03.429422 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.429291 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerName="config-reloader" Jul 07 17:57:03.429422 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.429296 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerName="kube-rbac-proxy" Jul 07 17:57:03.429422 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.429301 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerName="kube-rbac-proxy" Jul 07 17:57:03.429422 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.429309 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerName="alertmanager" Jul 07 17:57:03.429422 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.429314 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerName="alertmanager" Jul 07 17:57:03.429422 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.429326 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerName="init-config-reloader" Jul 07 17:57:03.429422 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.429331 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerName="init-config-reloader" Jul 07 17:57:03.429422 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.429383 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerName="kube-rbac-proxy-metric" Jul 07 17:57:03.429422 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.429393 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerName="alertmanager" Jul 07 17:57:03.429422 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.429399 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerName="kube-rbac-proxy" Jul 07 17:57:03.429422 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.429405 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerName="prom-label-proxy" Jul 07 17:57:03.429422 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.429410 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerName="kube-rbac-proxy-web" Jul 07 17:57:03.429422 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.429415 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0a24e05-c4bb-4beb-b46e-ec0e58d0832e" containerName="config-reloader" Jul 07 17:57:03.433235 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.433220 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.436128 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.436107 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 07 17:57:03.436230 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.436149 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 07 17:57:03.436230 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.436159 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 07 17:57:03.436350 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.436332 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 07 17:57:03.436461 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.436445 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 07 17:57:03.436574 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.436559 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 07 17:57:03.436734 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.436722 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 07 17:57:03.436983 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.436967 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 07 17:57:03.437273 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.437258 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-m2487\"" Jul 07 17:57:03.444162 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.444142 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 07 17:57:03.447877 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.447855 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 17:57:03.507087 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.507046 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b776ec07-9a59-4ed8-9f3a-7880c377d46a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.507262 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.507103 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b776ec07-9a59-4ed8-9f3a-7880c377d46a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.507262 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.507143 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b776ec07-9a59-4ed8-9f3a-7880c377d46a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.507262 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.507173 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b776ec07-9a59-4ed8-9f3a-7880c377d46a-web-config\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.507262 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.507205 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldlg9\" (UniqueName: \"kubernetes.io/projected/b776ec07-9a59-4ed8-9f3a-7880c377d46a-kube-api-access-ldlg9\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.507262 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.507234 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b776ec07-9a59-4ed8-9f3a-7880c377d46a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.507262 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.507249 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b776ec07-9a59-4ed8-9f3a-7880c377d46a-config-volume\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.507564 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.507269 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b776ec07-9a59-4ed8-9f3a-7880c377d46a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.507564 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.507291 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b776ec07-9a59-4ed8-9f3a-7880c377d46a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.507564 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.507370 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b776ec07-9a59-4ed8-9f3a-7880c377d46a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.507564 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.507411 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b776ec07-9a59-4ed8-9f3a-7880c377d46a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.507564 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.507440 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b776ec07-9a59-4ed8-9f3a-7880c377d46a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.507564 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.507496 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b776ec07-9a59-4ed8-9f3a-7880c377d46a-config-out\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.608663 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.608559 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b776ec07-9a59-4ed8-9f3a-7880c377d46a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.608663 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.608603 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b776ec07-9a59-4ed8-9f3a-7880c377d46a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.608663 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.608632 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b776ec07-9a59-4ed8-9f3a-7880c377d46a-web-config\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.608663 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.608657 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldlg9\" (UniqueName: \"kubernetes.io/projected/b776ec07-9a59-4ed8-9f3a-7880c377d46a-kube-api-access-ldlg9\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.609075 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.608697 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b776ec07-9a59-4ed8-9f3a-7880c377d46a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.609075 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.608719 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b776ec07-9a59-4ed8-9f3a-7880c377d46a-config-volume\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.609075 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.608738 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b776ec07-9a59-4ed8-9f3a-7880c377d46a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.609075 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.608761 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b776ec07-9a59-4ed8-9f3a-7880c377d46a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.609275 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.609107 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b776ec07-9a59-4ed8-9f3a-7880c377d46a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.609275 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.609154 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b776ec07-9a59-4ed8-9f3a-7880c377d46a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.609275 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.609188 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b776ec07-9a59-4ed8-9f3a-7880c377d46a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.609275 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.609221 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b776ec07-9a59-4ed8-9f3a-7880c377d46a-config-out\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.609482 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.609336 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b776ec07-9a59-4ed8-9f3a-7880c377d46a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.609741 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.609712 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b776ec07-9a59-4ed8-9f3a-7880c377d46a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.610231 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.610208 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b776ec07-9a59-4ed8-9f3a-7880c377d46a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.612029 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.612002 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b776ec07-9a59-4ed8-9f3a-7880c377d46a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.612029 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.612024 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b776ec07-9a59-4ed8-9f3a-7880c377d46a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.612196 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.612089 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b776ec07-9a59-4ed8-9f3a-7880c377d46a-web-config\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.612196 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.612160 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b776ec07-9a59-4ed8-9f3a-7880c377d46a-config-volume\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.612300 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.612251 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b776ec07-9a59-4ed8-9f3a-7880c377d46a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.612300 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.612289 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b776ec07-9a59-4ed8-9f3a-7880c377d46a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.612569 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.612491 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b776ec07-9a59-4ed8-9f3a-7880c377d46a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.612772 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.612762 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b776ec07-9a59-4ed8-9f3a-7880c377d46a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.612845 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.612765 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b776ec07-9a59-4ed8-9f3a-7880c377d46a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.613915 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.613884 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b776ec07-9a59-4ed8-9f3a-7880c377d46a-config-out\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.618089 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.618073 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldlg9\" (UniqueName: \"kubernetes.io/projected/b776ec07-9a59-4ed8-9f3a-7880c377d46a-kube-api-access-ldlg9\") pod \"alertmanager-main-0\" (UID: \"b776ec07-9a59-4ed8-9f3a-7880c377d46a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.743536 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.743496 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 17:57:03.888632 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:03.888525 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 17:57:03.891558 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:57:03.891532 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb776ec07_9a59_4ed8_9f3a_7880c377d46a.slice/crio-ed855d6f6e63d47cecf5d0efd2c0269a25426f31a95cde22bb995a1e0ce7577d WatchSource:0}: Error finding container ed855d6f6e63d47cecf5d0efd2c0269a25426f31a95cde22bb995a1e0ce7577d: Status 404 returned error can't find the container with id ed855d6f6e63d47cecf5d0efd2c0269a25426f31a95cde22bb995a1e0ce7577d Jul 07 17:57:04.367332 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:04.367300 2583 generic.go:356] "Generic (PLEG): container finished" podID="b776ec07-9a59-4ed8-9f3a-7880c377d46a" containerID="bc7bb7b51518251e4fe0c7a3209e83a25058f72aa71c141c9da4bff1d1a3a737" exitCode=0 Jul 07 17:57:04.367758 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:04.367376 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b776ec07-9a59-4ed8-9f3a-7880c377d46a","Type":"ContainerDied","Data":"bc7bb7b51518251e4fe0c7a3209e83a25058f72aa71c141c9da4bff1d1a3a737"} Jul 07 17:57:04.367758 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:04.367451 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b776ec07-9a59-4ed8-9f3a-7880c377d46a","Type":"ContainerStarted","Data":"ed855d6f6e63d47cecf5d0efd2c0269a25426f31a95cde22bb995a1e0ce7577d"} Jul 07 17:57:05.376050 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:05.376016 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b776ec07-9a59-4ed8-9f3a-7880c377d46a","Type":"ContainerStarted","Data":"4772150c6a061957906e95c5bc50ea9bba32ba30134a2a225e46df070ef58ee5"} Jul 07 17:57:05.376432 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:05.376112 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b776ec07-9a59-4ed8-9f3a-7880c377d46a","Type":"ContainerStarted","Data":"8ef04715376a4f74c5cd516575c101af24c79386d82ed01e7d2611cec9920f1b"} Jul 07 17:57:05.376432 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:05.376139 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b776ec07-9a59-4ed8-9f3a-7880c377d46a","Type":"ContainerStarted","Data":"5c7aab84c2a8763c8c3956487d5a04681a0e9d498c0019cd15e1dcb85bc654f0"} Jul 07 17:57:05.376432 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:05.376163 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b776ec07-9a59-4ed8-9f3a-7880c377d46a","Type":"ContainerStarted","Data":"331fe0c6b74f7680dae435fbde2a685cd5f91456e241347d6762468c786fb18b"} Jul 07 17:57:05.376432 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:05.376187 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b776ec07-9a59-4ed8-9f3a-7880c377d46a","Type":"ContainerStarted","Data":"170fe5468dc658a30491a8f716e7acb69c5d639991c774811b6a6c61ce8e2730"} Jul 07 17:57:05.376432 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:05.376210 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b776ec07-9a59-4ed8-9f3a-7880c377d46a","Type":"ContainerStarted","Data":"07adf0d5f9e0bbe54ca78d209bd1dbdc4ea260f166cf629710566bc9d9d6167b"} Jul 07 17:57:05.408452 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:05.408379 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.394050323 podStartE2EDuration="2.408363078s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.312755ms" isStatefulPod=false podCreationTimestamp="2026-07-07 17:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:57:05.407295194 +0000 UTC m=+174.489898898" watchObservedRunningTime="2026-07-07 17:57:05.408363078 +0000 UTC m=+174.490966785" Jul 07 17:57:06.406572 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.406541 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 17:57:06.407046 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.406815 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerName="prometheus" containerID="cri-o://7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d" gracePeriod=600 Jul 07 17:57:06.407046 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.406841 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerName="kube-rbac-proxy-web" containerID="cri-o://d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf" gracePeriod=600 Jul 07 17:57:06.407046 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.406876 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerName="kube-rbac-proxy" containerID="cri-o://44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b" gracePeriod=600 Jul 07 17:57:06.407046 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.406870 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerName="kube-rbac-proxy-thanos" containerID="cri-o://053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd" gracePeriod=600 Jul 07 17:57:06.407046 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.406874 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerName="thanos-sidecar" containerID="cri-o://5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53" gracePeriod=600 Jul 07 17:57:06.407321 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.407026 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerName="config-reloader" containerID="cri-o://4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b" gracePeriod=600 Jul 07 17:57:06.659558 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.659494 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:06.738510 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.738479 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-configmap-metrics-client-ca\") pod \"25545dec-41dd-4edf-9906-12bc2d5be5b2\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " Jul 07 17:57:06.738510 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.738517 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-prometheus-k8s-rulefiles-2\") pod \"25545dec-41dd-4edf-9906-12bc2d5be5b2\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " Jul 07 17:57:06.738726 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.738544 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-prometheus-trusted-ca-bundle\") pod \"25545dec-41dd-4edf-9906-12bc2d5be5b2\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " Jul 07 17:57:06.738726 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.738571 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"25545dec-41dd-4edf-9906-12bc2d5be5b2\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " Jul 07 17:57:06.738726 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.738595 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-metrics-client-certs\") pod \"25545dec-41dd-4edf-9906-12bc2d5be5b2\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " Jul 07 17:57:06.738726 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.738706 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25545dec-41dd-4edf-9906-12bc2d5be5b2-config-out\") pod \"25545dec-41dd-4edf-9906-12bc2d5be5b2\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " Jul 07 17:57:06.738998 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.738963 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "25545dec-41dd-4edf-9906-12bc2d5be5b2" (UID: "25545dec-41dd-4edf-9906-12bc2d5be5b2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:57:06.738998 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.738986 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "25545dec-41dd-4edf-9906-12bc2d5be5b2" (UID: "25545dec-41dd-4edf-9906-12bc2d5be5b2"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:57:06.739143 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.739055 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "25545dec-41dd-4edf-9906-12bc2d5be5b2" (UID: "25545dec-41dd-4edf-9906-12bc2d5be5b2"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:57:06.739235 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.738795 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-prometheus-k8s-rulefiles-0\") pod \"25545dec-41dd-4edf-9906-12bc2d5be5b2\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " Jul 07 17:57:06.739529 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.739259 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"25545dec-41dd-4edf-9906-12bc2d5be5b2\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " Jul 07 17:57:06.739529 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.739297 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-web-config\") pod \"25545dec-41dd-4edf-9906-12bc2d5be5b2\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " Jul 07 17:57:06.739529 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.739333 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-thanos-prometheus-http-client-file\") pod \"25545dec-41dd-4edf-9906-12bc2d5be5b2\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " Jul 07 17:57:06.739729 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.739707 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25545dec-41dd-4edf-9906-12bc2d5be5b2-tls-assets\") pod \"25545dec-41dd-4edf-9906-12bc2d5be5b2\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " Jul 07 17:57:06.739805 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.739763 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-prometheus-k8s-tls\") pod \"25545dec-41dd-4edf-9906-12bc2d5be5b2\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " Jul 07 17:57:06.739805 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.739800 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kr9k6\" (UniqueName: \"kubernetes.io/projected/25545dec-41dd-4edf-9906-12bc2d5be5b2-kube-api-access-kr9k6\") pod \"25545dec-41dd-4edf-9906-12bc2d5be5b2\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " Jul 07 17:57:06.740817 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.739858 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-prometheus-k8s-rulefiles-1\") pod \"25545dec-41dd-4edf-9906-12bc2d5be5b2\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " Jul 07 17:57:06.740817 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.739886 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-configmap-serving-certs-ca-bundle\") pod \"25545dec-41dd-4edf-9906-12bc2d5be5b2\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " Jul 07 17:57:06.740817 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.739928 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-configmap-kubelet-serving-ca-bundle\") pod \"25545dec-41dd-4edf-9906-12bc2d5be5b2\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " Jul 07 17:57:06.740817 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.739957 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-grpc-tls\") pod \"25545dec-41dd-4edf-9906-12bc2d5be5b2\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " Jul 07 17:57:06.740817 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.739981 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/25545dec-41dd-4edf-9906-12bc2d5be5b2-prometheus-k8s-db\") pod \"25545dec-41dd-4edf-9906-12bc2d5be5b2\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " Jul 07 17:57:06.740817 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.740023 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-kube-rbac-proxy\") pod \"25545dec-41dd-4edf-9906-12bc2d5be5b2\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " Jul 07 17:57:06.740817 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.740057 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-config\") pod \"25545dec-41dd-4edf-9906-12bc2d5be5b2\" (UID: \"25545dec-41dd-4edf-9906-12bc2d5be5b2\") " Jul 07 17:57:06.740817 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.740348 2583 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-configmap-metrics-client-ca\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:06.740817 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.740368 2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:06.740817 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.740385 2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:06.740817 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.740537 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "25545dec-41dd-4edf-9906-12bc2d5be5b2" (UID: "25545dec-41dd-4edf-9906-12bc2d5be5b2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:57:06.740817 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.740720 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "25545dec-41dd-4edf-9906-12bc2d5be5b2" (UID: "25545dec-41dd-4edf-9906-12bc2d5be5b2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:57:06.741619 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.741131 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "25545dec-41dd-4edf-9906-12bc2d5be5b2" (UID: "25545dec-41dd-4edf-9906-12bc2d5be5b2"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:57:06.741619 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.741641 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "25545dec-41dd-4edf-9906-12bc2d5be5b2" (UID: "25545dec-41dd-4edf-9906-12bc2d5be5b2"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:57:06.741816 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.741786 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "25545dec-41dd-4edf-9906-12bc2d5be5b2" (UID: "25545dec-41dd-4edf-9906-12bc2d5be5b2"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:57:06.742051 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.742019 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "25545dec-41dd-4edf-9906-12bc2d5be5b2" (UID: "25545dec-41dd-4edf-9906-12bc2d5be5b2"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 17:57:06.742716 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.742684 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25545dec-41dd-4edf-9906-12bc2d5be5b2-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "25545dec-41dd-4edf-9906-12bc2d5be5b2" (UID: "25545dec-41dd-4edf-9906-12bc2d5be5b2"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:57:06.743535 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.743449 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "25545dec-41dd-4edf-9906-12bc2d5be5b2" (UID: "25545dec-41dd-4edf-9906-12bc2d5be5b2"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:57:06.744185 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.744164 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "25545dec-41dd-4edf-9906-12bc2d5be5b2" (UID: "25545dec-41dd-4edf-9906-12bc2d5be5b2"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:57:06.744385 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.744362 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "25545dec-41dd-4edf-9906-12bc2d5be5b2" (UID: "25545dec-41dd-4edf-9906-12bc2d5be5b2"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:57:06.744385 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.744377 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25545dec-41dd-4edf-9906-12bc2d5be5b2-kube-api-access-kr9k6" (OuterVolumeSpecName: "kube-api-access-kr9k6") pod "25545dec-41dd-4edf-9906-12bc2d5be5b2" (UID: "25545dec-41dd-4edf-9906-12bc2d5be5b2"). InnerVolumeSpecName "kube-api-access-kr9k6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:57:06.744537 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.744449 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25545dec-41dd-4edf-9906-12bc2d5be5b2-config-out" (OuterVolumeSpecName: "config-out") pod "25545dec-41dd-4edf-9906-12bc2d5be5b2" (UID: "25545dec-41dd-4edf-9906-12bc2d5be5b2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 17:57:06.744537 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.744451 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "25545dec-41dd-4edf-9906-12bc2d5be5b2" (UID: "25545dec-41dd-4edf-9906-12bc2d5be5b2"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:57:06.745331 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.745292 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-config" (OuterVolumeSpecName: "config") pod "25545dec-41dd-4edf-9906-12bc2d5be5b2" (UID: "25545dec-41dd-4edf-9906-12bc2d5be5b2"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:57:06.745588 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.745567 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25545dec-41dd-4edf-9906-12bc2d5be5b2-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "25545dec-41dd-4edf-9906-12bc2d5be5b2" (UID: "25545dec-41dd-4edf-9906-12bc2d5be5b2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 17:57:06.745873 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.745850 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "25545dec-41dd-4edf-9906-12bc2d5be5b2" (UID: "25545dec-41dd-4edf-9906-12bc2d5be5b2"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:57:06.755467 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.755438 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-web-config" (OuterVolumeSpecName: "web-config") pod "25545dec-41dd-4edf-9906-12bc2d5be5b2" (UID: "25545dec-41dd-4edf-9906-12bc2d5be5b2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 17:57:06.841776 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.841741 2583 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25545dec-41dd-4edf-9906-12bc2d5be5b2-config-out\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:06.841776 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.841769 2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:06.841776 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.841781 2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:06.842029 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.841790 2583 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-web-config\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:06.842029 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.841799 2583 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:06.842029 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.841809 2583 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25545dec-41dd-4edf-9906-12bc2d5be5b2-tls-assets\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:06.842029 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.841817 2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:06.842029 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.841825 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr9k6\" (UniqueName: \"kubernetes.io/projected/25545dec-41dd-4edf-9906-12bc2d5be5b2-kube-api-access-kr9k6\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:06.842029 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.841834 2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:06.842029 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.841843 2583 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:06.842029 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.841851 2583 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25545dec-41dd-4edf-9906-12bc2d5be5b2-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:06.842029 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.841860 2583 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-grpc-tls\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:06.842029 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.841868 2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/25545dec-41dd-4edf-9906-12bc2d5be5b2-prometheus-k8s-db\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:06.842029 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.841876 2583 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-kube-rbac-proxy\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:06.842029 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.841883 2583 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-config\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:06.842029 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.841891 2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:06.842029 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:06.841930 2583 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/25545dec-41dd-4edf-9906-12bc2d5be5b2-secret-metrics-client-certs\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 17:57:07.390186 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.390149 2583 generic.go:356] "Generic (PLEG): container finished" podID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerID="053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd" exitCode=0 Jul 07 17:57:07.390186 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.390175 2583 generic.go:356] "Generic (PLEG): container finished" podID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerID="44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b" exitCode=0 Jul 07 17:57:07.390186 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.390183 2583 generic.go:356] "Generic (PLEG): container finished" podID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerID="d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf" exitCode=0 Jul 07 17:57:07.390186 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.390190 2583 generic.go:356] "Generic (PLEG): container finished" podID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerID="5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53" exitCode=0 Jul 07 17:57:07.390186 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.390197 2583 generic.go:356] "Generic (PLEG): container finished" podID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerID="4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b" exitCode=0 Jul 07 17:57:07.390580 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.390202 2583 generic.go:356] "Generic (PLEG): container finished" podID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerID="7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d" exitCode=0 Jul 07 17:57:07.390580 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.390233 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25545dec-41dd-4edf-9906-12bc2d5be5b2","Type":"ContainerDied","Data":"053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd"} Jul 07 17:57:07.390580 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.390254 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.390580 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.390300 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25545dec-41dd-4edf-9906-12bc2d5be5b2","Type":"ContainerDied","Data":"44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b"} Jul 07 17:57:07.390580 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.390328 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25545dec-41dd-4edf-9906-12bc2d5be5b2","Type":"ContainerDied","Data":"d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf"} Jul 07 17:57:07.390580 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.390356 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25545dec-41dd-4edf-9906-12bc2d5be5b2","Type":"ContainerDied","Data":"5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53"} Jul 07 17:57:07.390580 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.390369 2583 scope.go:117] "RemoveContainer" containerID="053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd" Jul 07 17:57:07.390580 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.390388 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25545dec-41dd-4edf-9906-12bc2d5be5b2","Type":"ContainerDied","Data":"4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b"} Jul 07 17:57:07.390580 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.390420 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25545dec-41dd-4edf-9906-12bc2d5be5b2","Type":"ContainerDied","Data":"7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d"} Jul 07 17:57:07.390580 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.390446 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25545dec-41dd-4edf-9906-12bc2d5be5b2","Type":"ContainerDied","Data":"e03c997ec421a0b686e40e8da32d0e87b9b1aa5402a5799bdb10a38b0e698b5a"} Jul 07 17:57:07.404123 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.399288 2583 scope.go:117] "RemoveContainer" containerID="44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b" Jul 07 17:57:07.408922 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.408735 2583 scope.go:117] "RemoveContainer" containerID="d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf" Jul 07 17:57:07.416052 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.416032 2583 scope.go:117] "RemoveContainer" containerID="5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53" Jul 07 17:57:07.422623 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.422602 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 17:57:07.430235 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.430208 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 17:57:07.438940 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.438888 2583 scope.go:117] "RemoveContainer" containerID="4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b" Jul 07 17:57:07.446723 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.446703 2583 scope.go:117] "RemoveContainer" containerID="7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d" Jul 07 17:57:07.454801 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.454778 2583 scope.go:117] "RemoveContainer" containerID="d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1" Jul 07 17:57:07.461883 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.461856 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 17:57:07.462206 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.462192 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerName="kube-rbac-proxy-web" Jul 07 17:57:07.462259 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.462208 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerName="kube-rbac-proxy-web" Jul 07 17:57:07.462259 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.462217 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerName="kube-rbac-proxy-thanos" Jul 07 17:57:07.462259 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.462223 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerName="kube-rbac-proxy-thanos" Jul 07 17:57:07.462259 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.462232 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerName="config-reloader" Jul 07 17:57:07.462259 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.462238 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerName="config-reloader" Jul 07 17:57:07.462259 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.462247 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerName="thanos-sidecar" Jul 07 17:57:07.462259 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.462252 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerName="thanos-sidecar" Jul 07 17:57:07.462476 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.462263 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerName="kube-rbac-proxy" Jul 07 17:57:07.462476 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.462270 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerName="kube-rbac-proxy" Jul 07 17:57:07.462476 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.462279 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerName="init-config-reloader" Jul 07 17:57:07.462476 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.462285 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerName="init-config-reloader" Jul 07 17:57:07.462476 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.462291 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerName="prometheus" Jul 07 17:57:07.462476 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.462296 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerName="prometheus" Jul 07 17:57:07.462476 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.462341 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerName="kube-rbac-proxy" Jul 07 17:57:07.462476 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.462348 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerName="kube-rbac-proxy-web" Jul 07 17:57:07.462476 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.462357 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerName="prometheus" Jul 07 17:57:07.462476 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.462364 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerName="thanos-sidecar" Jul 07 17:57:07.462476 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.462369 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerName="kube-rbac-proxy-thanos" Jul 07 17:57:07.462476 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.462376 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="25545dec-41dd-4edf-9906-12bc2d5be5b2" containerName="config-reloader" Jul 07 17:57:07.463431 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.463403 2583 scope.go:117] "RemoveContainer" containerID="053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd" Jul 07 17:57:07.463735 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:57:07.463713 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd\": container with ID starting with 053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd not found: ID does not exist" containerID="053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd" Jul 07 17:57:07.463800 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.463744 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd"} err="failed to get container status \"053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd\": rpc error: code = NotFound desc = could not find container \"053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd\": container with ID starting with 053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd not found: ID does not exist" Jul 07 17:57:07.463800 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.463762 2583 scope.go:117] "RemoveContainer" containerID="44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b" Jul 07 17:57:07.464057 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:57:07.464039 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b\": container with ID starting with 44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b not found: ID does not exist" containerID="44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b" Jul 07 17:57:07.464103 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.464065 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b"} err="failed to get container status \"44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b\": rpc error: code = NotFound desc = could not find container \"44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b\": container with ID starting with 44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b not found: ID does not exist" Jul 07 17:57:07.464103 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.464082 2583 scope.go:117] "RemoveContainer" containerID="d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf" Jul 07 17:57:07.464274 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:57:07.464256 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf\": container with ID starting with d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf not found: ID does not exist" containerID="d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf" Jul 07 17:57:07.464315 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.464280 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf"} err="failed to get container status \"d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf\": rpc error: code = NotFound desc = could not find container \"d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf\": container with ID starting with d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf not found: ID does not exist" Jul 07 17:57:07.464315 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.464294 2583 scope.go:117] "RemoveContainer" containerID="5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53" Jul 07 17:57:07.464529 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:57:07.464516 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53\": container with ID starting with 5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53 not found: ID does not exist" containerID="5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53" Jul 07 17:57:07.464574 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.464531 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53"} err="failed to get container status \"5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53\": rpc error: code = NotFound desc = could not find container \"5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53\": container with ID starting with 5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53 not found: ID does not exist" Jul 07 17:57:07.464574 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.464543 2583 scope.go:117] "RemoveContainer" containerID="4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b" Jul 07 17:57:07.464715 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:57:07.464696 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b\": container with ID starting with 4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b not found: ID does not exist" containerID="4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b" Jul 07 17:57:07.464754 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.464723 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b"} err="failed to get container status \"4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b\": rpc error: code = NotFound desc = could not find container \"4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b\": container with ID starting with 4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b not found: ID does not exist" Jul 07 17:57:07.464754 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.464742 2583 scope.go:117] "RemoveContainer" containerID="7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d" Jul 07 17:57:07.464951 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:57:07.464936 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d\": container with ID starting with 7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d not found: ID does not exist" containerID="7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d" Jul 07 17:57:07.465011 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.464954 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d"} err="failed to get container status \"7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d\": rpc error: code = NotFound desc = could not find container \"7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d\": container with ID starting with 7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d not found: ID does not exist" Jul 07 17:57:07.465011 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.464966 2583 scope.go:117] "RemoveContainer" containerID="d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1" Jul 07 17:57:07.465235 ip-10-0-136-192 kubenswrapper[2583]: E0707 17:57:07.465207 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1\": container with ID starting with d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1 not found: ID does not exist" containerID="d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1" Jul 07 17:57:07.465308 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.465230 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1"} err="failed to get container status \"d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1\": rpc error: code = NotFound desc = could not find container \"d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1\": container with ID starting with d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1 not found: ID does not exist" Jul 07 17:57:07.465308 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.465250 2583 scope.go:117] "RemoveContainer" containerID="053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd" Jul 07 17:57:07.465506 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.465488 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd"} err="failed to get container status \"053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd\": rpc error: code = NotFound desc = could not find container \"053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd\": container with ID starting with 053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd not found: ID does not exist" Jul 07 17:57:07.465558 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.465507 2583 scope.go:117] "RemoveContainer" containerID="44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b" Jul 07 17:57:07.465759 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.465740 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b"} err="failed to get container status \"44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b\": rpc error: code = NotFound desc = could not find container \"44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b\": container with ID starting with 44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b not found: ID does not exist" Jul 07 17:57:07.465830 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.465761 2583 scope.go:117] "RemoveContainer" containerID="d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf" Jul 07 17:57:07.466055 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.466038 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf"} err="failed to get container status \"d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf\": rpc error: code = NotFound desc = could not find container \"d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf\": container with ID starting with d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf not found: ID does not exist" Jul 07 17:57:07.466115 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.466057 2583 scope.go:117] "RemoveContainer" containerID="5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53" Jul 07 17:57:07.466282 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.466264 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53"} err="failed to get container status \"5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53\": rpc error: code = NotFound desc = could not find container \"5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53\": container with ID starting with 5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53 not found: ID does not exist" Jul 07 17:57:07.466321 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.466283 2583 scope.go:117] "RemoveContainer" containerID="4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b" Jul 07 17:57:07.466501 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.466483 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b"} err="failed to get container status \"4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b\": rpc error: code = NotFound desc = could not find container \"4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b\": container with ID starting with 4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b not found: ID does not exist" Jul 07 17:57:07.466569 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.466504 2583 scope.go:117] "RemoveContainer" containerID="7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d" Jul 07 17:57:07.466697 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.466681 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d"} err="failed to get container status \"7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d\": rpc error: code = NotFound desc = could not find container \"7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d\": container with ID starting with 7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d not found: ID does not exist" Jul 07 17:57:07.466754 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.466698 2583 scope.go:117] "RemoveContainer" containerID="d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1" Jul 07 17:57:07.466947 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.466922 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1"} err="failed to get container status \"d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1\": rpc error: code = NotFound desc = could not find container \"d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1\": container with ID starting with d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1 not found: ID does not exist" Jul 07 17:57:07.466947 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.466941 2583 scope.go:117] "RemoveContainer" containerID="053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd" Jul 07 17:57:07.467150 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.467130 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd"} err="failed to get container status \"053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd\": rpc error: code = NotFound desc = could not find container \"053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd\": container with ID starting with 053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd not found: ID does not exist" Jul 07 17:57:07.467206 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.467152 2583 scope.go:117] "RemoveContainer" containerID="44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b" Jul 07 17:57:07.467387 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.467370 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b"} err="failed to get container status \"44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b\": rpc error: code = NotFound desc = could not find container \"44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b\": container with ID starting with 44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b not found: ID does not exist" Jul 07 17:57:07.467387 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.467387 2583 scope.go:117] "RemoveContainer" containerID="d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf" Jul 07 17:57:07.467596 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.467581 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.467701 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.467584 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf"} err="failed to get container status \"d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf\": rpc error: code = NotFound desc = could not find container \"d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf\": container with ID starting with d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf not found: ID does not exist" Jul 07 17:57:07.467792 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.467703 2583 scope.go:117] "RemoveContainer" containerID="5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53" Jul 07 17:57:07.468182 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.468150 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53"} err="failed to get container status \"5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53\": rpc error: code = NotFound desc = could not find container \"5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53\": container with ID starting with 5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53 not found: ID does not exist" Jul 07 17:57:07.468182 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.468183 2583 scope.go:117] "RemoveContainer" containerID="4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b" Jul 07 17:57:07.468454 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.468432 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b"} err="failed to get container status \"4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b\": rpc error: code = NotFound desc = could not find container \"4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b\": container with ID starting with 4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b not found: ID does not exist" Jul 07 17:57:07.468493 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.468457 2583 scope.go:117] "RemoveContainer" containerID="7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d" Jul 07 17:57:07.468700 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.468679 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d"} err="failed to get container status \"7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d\": rpc error: code = NotFound desc = could not find container \"7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d\": container with ID starting with 7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d not found: ID does not exist" Jul 07 17:57:07.468777 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.468702 2583 scope.go:117] "RemoveContainer" containerID="d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1" Jul 07 17:57:07.468956 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.468938 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1"} err="failed to get container status \"d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1\": rpc error: code = NotFound desc = could not find container \"d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1\": container with ID starting with d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1 not found: ID does not exist" Jul 07 17:57:07.469003 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.468958 2583 scope.go:117] "RemoveContainer" containerID="053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd" Jul 07 17:57:07.469165 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.469149 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd"} err="failed to get container status \"053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd\": rpc error: code = NotFound desc = could not find container \"053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd\": container with ID starting with 053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd not found: ID does not exist" Jul 07 17:57:07.469219 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.469165 2583 scope.go:117] "RemoveContainer" containerID="44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b" Jul 07 17:57:07.469382 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.469362 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b"} err="failed to get container status \"44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b\": rpc error: code = NotFound desc = could not find container \"44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b\": container with ID starting with 44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b not found: ID does not exist" Jul 07 17:57:07.469423 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.469385 2583 scope.go:117] "RemoveContainer" containerID="d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf" Jul 07 17:57:07.469624 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.469607 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf"} err="failed to get container status \"d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf\": rpc error: code = NotFound desc = could not find container \"d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf\": container with ID starting with d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf not found: ID does not exist" Jul 07 17:57:07.469624 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.469624 2583 scope.go:117] "RemoveContainer" containerID="5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53" Jul 07 17:57:07.469844 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.469820 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53"} err="failed to get container status \"5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53\": rpc error: code = NotFound desc = could not find container \"5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53\": container with ID starting with 5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53 not found: ID does not exist" Jul 07 17:57:07.469891 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.469847 2583 scope.go:117] "RemoveContainer" containerID="4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b" Jul 07 17:57:07.470097 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.470073 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b"} err="failed to get container status \"4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b\": rpc error: code = NotFound desc = could not find container \"4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b\": container with ID starting with 4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b not found: ID does not exist" Jul 07 17:57:07.470149 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.470098 2583 scope.go:117] "RemoveContainer" containerID="7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d" Jul 07 17:57:07.470313 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.470295 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d"} err="failed to get container status \"7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d\": rpc error: code = NotFound desc = could not find container \"7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d\": container with ID starting with 7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d not found: ID does not exist" Jul 07 17:57:07.470358 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.470313 2583 scope.go:117] "RemoveContainer" containerID="d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1" Jul 07 17:57:07.470544 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.470524 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1"} err="failed to get container status \"d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1\": rpc error: code = NotFound desc = could not find container \"d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1\": container with ID starting with d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1 not found: ID does not exist" Jul 07 17:57:07.470588 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.470545 2583 scope.go:117] "RemoveContainer" containerID="053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd" Jul 07 17:57:07.470756 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.470741 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd"} err="failed to get container status \"053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd\": rpc error: code = NotFound desc = could not find container \"053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd\": container with ID starting with 053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd not found: ID does not exist" Jul 07 17:57:07.470756 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.470756 2583 scope.go:117] "RemoveContainer" containerID="44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b" Jul 07 17:57:07.470983 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.470964 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b"} err="failed to get container status \"44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b\": rpc error: code = NotFound desc = could not find container \"44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b\": container with ID starting with 44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b not found: ID does not exist" Jul 07 17:57:07.471024 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.470983 2583 scope.go:117] "RemoveContainer" containerID="d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf" Jul 07 17:57:07.471175 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.471160 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf"} err="failed to get container status \"d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf\": rpc error: code = NotFound desc = could not find container \"d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf\": container with ID starting with d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf not found: ID does not exist" Jul 07 17:57:07.471219 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.471175 2583 scope.go:117] "RemoveContainer" containerID="5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53" Jul 07 17:57:07.471409 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.471390 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53"} err="failed to get container status \"5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53\": rpc error: code = NotFound desc = could not find container \"5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53\": container with ID starting with 5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53 not found: ID does not exist" Jul 07 17:57:07.471473 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.471410 2583 scope.go:117] "RemoveContainer" containerID="4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b" Jul 07 17:57:07.471641 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.471619 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b"} err="failed to get container status \"4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b\": rpc error: code = NotFound desc = could not find container \"4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b\": container with ID starting with 4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b not found: ID does not exist" Jul 07 17:57:07.471641 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.471641 2583 scope.go:117] "RemoveContainer" containerID="7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d" Jul 07 17:57:07.471855 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.471836 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d"} err="failed to get container status \"7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d\": rpc error: code = NotFound desc = could not find container \"7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d\": container with ID starting with 7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d not found: ID does not exist" Jul 07 17:57:07.471964 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.471855 2583 scope.go:117] "RemoveContainer" containerID="d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1" Jul 07 17:57:07.472075 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.472057 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1"} err="failed to get container status \"d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1\": rpc error: code = NotFound desc = could not find container \"d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1\": container with ID starting with d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1 not found: ID does not exist" Jul 07 17:57:07.472126 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.472076 2583 scope.go:117] "RemoveContainer" containerID="053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd" Jul 07 17:57:07.472302 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.472284 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd"} err="failed to get container status \"053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd\": rpc error: code = NotFound desc = could not find container \"053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd\": container with ID starting with 053732a1219aebb1f53846b78df1816212c7e96d65e2fd163625adafda4aaecd not found: ID does not exist" Jul 07 17:57:07.472350 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.472303 2583 scope.go:117] "RemoveContainer" containerID="44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b" Jul 07 17:57:07.472538 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.472522 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b"} err="failed to get container status \"44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b\": rpc error: code = NotFound desc = could not find container \"44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b\": container with ID starting with 44641956ab8097056b6ed31063bdbadd8597b43beac559b6de8ea4b34df86d7b not found: ID does not exist" Jul 07 17:57:07.472598 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.472539 2583 scope.go:117] "RemoveContainer" containerID="d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf" Jul 07 17:57:07.472796 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.472773 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf"} err="failed to get container status \"d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf\": rpc error: code = NotFound desc = could not find container \"d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf\": container with ID starting with d574f7823ee77ce18cb7a4e2bcddb064dc52befe44d722ff0cde7fb473452aaf not found: ID does not exist" Jul 07 17:57:07.472841 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.472800 2583 scope.go:117] "RemoveContainer" containerID="5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53" Jul 07 17:57:07.473025 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.473008 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53"} err="failed to get container status \"5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53\": rpc error: code = NotFound desc = could not find container \"5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53\": container with ID starting with 5bb963472fd1291821e075617485635ae2d924401ac115256ab0cb84d19edb53 not found: ID does not exist" Jul 07 17:57:07.473064 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.473025 2583 scope.go:117] "RemoveContainer" containerID="4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b" Jul 07 17:57:07.473214 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.473196 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b"} err="failed to get container status \"4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b\": rpc error: code = NotFound desc = could not find container \"4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b\": container with ID starting with 4c61ebbc9282b375b9d1727eb5be3165da0e840e43abf32366983bf4749dde7b not found: ID does not exist" Jul 07 17:57:07.473265 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.473215 2583 scope.go:117] "RemoveContainer" containerID="7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d" Jul 07 17:57:07.473490 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.473472 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d"} err="failed to get container status \"7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d\": rpc error: code = NotFound desc = could not find container \"7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d\": container with ID starting with 7753667c63f6718d2a2b155e23223ca38290c54a4eb9e648dc4041551cb6cd1d not found: ID does not exist" Jul 07 17:57:07.473560 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.473490 2583 scope.go:117] "RemoveContainer" containerID="d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1" Jul 07 17:57:07.473692 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.473672 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1"} err="failed to get container status \"d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1\": rpc error: code = NotFound desc = could not find container \"d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1\": container with ID starting with d643b1028034008ad488ab335018513da71c8f36111561d4c647e911ce7fbbc1 not found: ID does not exist" Jul 07 17:57:07.475347 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.475320 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5j0pnigmdrias\"" Jul 07 17:57:07.475458 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.475380 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 07 17:57:07.476540 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.476520 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 07 17:57:07.476671 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.476579 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6v2jb\"" Jul 07 17:57:07.476671 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.476524 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 07 17:57:07.476671 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.476668 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 07 17:57:07.476849 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.476791 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 07 17:57:07.476923 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.476867 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 07 17:57:07.476974 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.476957 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 07 17:57:07.477151 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.477135 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 07 17:57:07.477383 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.477364 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 07 17:57:07.477487 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.477397 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 07 17:57:07.477487 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.477463 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 07 17:57:07.477678 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.477665 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 07 17:57:07.479284 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.479270 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 07 17:57:07.486616 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.486599 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 07 17:57:07.494845 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.494820 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 17:57:07.547757 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.547714 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-web-config\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.547993 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.547767 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.547993 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.547798 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-config-out\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.547993 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.547820 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.547993 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.547847 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.547993 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.547868 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.547993 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.547895 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.547993 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.547956 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.547993 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.547987 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.548425 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.548012 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.548425 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.548041 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.548425 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.548073 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.548425 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.548096 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.548425 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.548220 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.548425 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.548275 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-config\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.548425 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.548304 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.548425 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.548385 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.548740 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.548437 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.548740 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.548468 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.548740 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.548490 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n9wv\" (UniqueName: \"kubernetes.io/projected/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-kube-api-access-4n9wv\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.649262 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.649172 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.649262 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.649210 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-config\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.649262 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.649230 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.649262 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.649263 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.649552 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.649296 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.649552 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.649314 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.649552 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.649333 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n9wv\" (UniqueName: \"kubernetes.io/projected/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-kube-api-access-4n9wv\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.649552 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.649364 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-web-config\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.649552 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.649394 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.649552 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.649422 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-config-out\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.649552 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.649450 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.649552 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.649478 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.649552 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.649498 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.649552 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.649524 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.650516 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.649568 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.650516 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.649605 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.650516 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.649629 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.650516 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.649660 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.650516 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.649693 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.650516 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.649717 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.651757 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.650842 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.651757 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.650936 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.651757 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.651250 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.651757 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.651563 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.651998 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.650981 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.652258 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.652230 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.652722 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.652696 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.652804 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.652776 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.653469 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.653433 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.653801 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.653774 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.653870 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.653828 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-config\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.655164 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.655137 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.655238 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.655144 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.655238 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.655144 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.655431 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.655400 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.655637 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.655607 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.655637 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.655626 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-web-config\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.656424 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.656394 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.656593 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.656574 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-config-out\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.658786 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.658762 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n9wv\" (UniqueName: \"kubernetes.io/projected/2bb751ed-3254-44f6-ac51-d2f6246c6cd4-kube-api-access-4n9wv\") pod \"prometheus-k8s-0\" (UID: \"2bb751ed-3254-44f6-ac51-d2f6246c6cd4\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.778309 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.778268 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:07.940954 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:07.940846 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 17:57:07.943731 ip-10-0-136-192 kubenswrapper[2583]: W0707 17:57:07.943704 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb751ed_3254_44f6_ac51_d2f6246c6cd4.slice/crio-aa9c58d8002785727c64d2fe25c6abdf7d351b2f612137dbf378e41117e9e2da WatchSource:0}: Error finding container aa9c58d8002785727c64d2fe25c6abdf7d351b2f612137dbf378e41117e9e2da: Status 404 returned error can't find the container with id aa9c58d8002785727c64d2fe25c6abdf7d351b2f612137dbf378e41117e9e2da Jul 07 17:57:08.396967 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:08.396926 2583 generic.go:356] "Generic (PLEG): container finished" podID="2bb751ed-3254-44f6-ac51-d2f6246c6cd4" containerID="593f3814a17714cd8dc7a52cbe916c81e8610d3f657958f11db177fbe7869b0a" exitCode=0 Jul 07 17:57:08.397137 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:08.396972 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2bb751ed-3254-44f6-ac51-d2f6246c6cd4","Type":"ContainerDied","Data":"593f3814a17714cd8dc7a52cbe916c81e8610d3f657958f11db177fbe7869b0a"} Jul 07 17:57:08.397137 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:08.397038 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2bb751ed-3254-44f6-ac51-d2f6246c6cd4","Type":"ContainerStarted","Data":"aa9c58d8002785727c64d2fe25c6abdf7d351b2f612137dbf378e41117e9e2da"} Jul 07 17:57:09.405829 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:09.405792 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2bb751ed-3254-44f6-ac51-d2f6246c6cd4","Type":"ContainerStarted","Data":"37c1028cce0ce4fd2aa913dbc559c92ef6617d987dffd085c56ab787182cf456"} Jul 07 17:57:09.406213 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:09.405887 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2bb751ed-3254-44f6-ac51-d2f6246c6cd4","Type":"ContainerStarted","Data":"1cfbca9be2ba5a9b9f42985a0e53b7dbfd67d3f96ac2c0587e79fe8e48c5df6c"} Jul 07 17:57:09.406213 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:09.405948 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2bb751ed-3254-44f6-ac51-d2f6246c6cd4","Type":"ContainerStarted","Data":"48b8116fc2c0cf4d92220ee74a4ef3de183bef079376354036687d7ca6739077"} Jul 07 17:57:09.406213 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:09.405974 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2bb751ed-3254-44f6-ac51-d2f6246c6cd4","Type":"ContainerStarted","Data":"1bd8777e6e1a8fd1f677537ddef6d09b31d3f2ac2d8ca9e24eb01a178886395c"} Jul 07 17:57:09.406213 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:09.405998 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2bb751ed-3254-44f6-ac51-d2f6246c6cd4","Type":"ContainerStarted","Data":"c4f6504ece300d95c6e9c9f86bdf47ee3279accc7a9d2eca6edb1e7318692662"} Jul 07 17:57:09.406213 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:09.406021 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2bb751ed-3254-44f6-ac51-d2f6246c6cd4","Type":"ContainerStarted","Data":"ee30fbdbaea3be3922fff2dff9155cd763dc61f915c01bbbe15f308e0aa5b7a5"} Jul 07 17:57:09.428375 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:09.428342 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25545dec-41dd-4edf-9906-12bc2d5be5b2" path="/var/lib/kubelet/pods/25545dec-41dd-4edf-9906-12bc2d5be5b2/volumes" Jul 07 17:57:09.443346 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:09.443284 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.425869851 podStartE2EDuration="2.443262971s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.39312ms" isStatefulPod=false podCreationTimestamp="2026-07-07 17:57:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 17:57:09.442433657 +0000 UTC m=+178.525037361" watchObservedRunningTime="2026-07-07 17:57:09.443262971 +0000 UTC m=+178.525866677" Jul 07 17:57:11.476845 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:11.476796 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:57:12.779373 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:12.779330 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:57:22.656173 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:22.656129 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:57:22.755064 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:22.755032 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:57:22.855763 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:22.855710 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:57:34.604141 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:34.604107 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:57:38.746892 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:38.746854 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:57:46.447142 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:46.447104 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:57:55.760612 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:55.760573 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:57:55.843987 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:57:55.843946 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:07.779183 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:58:07.779137 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:58:07.796043 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:58:07.796010 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:58:08.668104 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:58:08.668074 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 17:58:16.237715 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:58:16.237673 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:16.352498 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:58:16.352467 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:28.213606 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:58:28.213542 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:28.316976 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:58:28.316940 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:32.187988 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:58:32.187876 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:32.294285 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:58:32.294253 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:32.408026 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:58:32.407982 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:37.158545 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:58:37.158505 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:37.283338 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:58:37.283299 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:37.408837 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:58:37.408746 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:37.935395 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:58:37.935351 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:38.024453 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:58:38.024411 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:45.849990 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:58:45.849955 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:47.960774 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:58:47.960736 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:58:48.125869 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:58:48.125832 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:59:11.297261 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:59:11.297232 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wqfm4_997df85c-7632-47b5-a8a6-1c37f3346251/console-operator/2.log" Jul 07 17:59:11.302453 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:59:11.302421 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wqfm4_997df85c-7632-47b5-a8a6-1c37f3346251/console-operator/2.log" Jul 07 17:59:11.304735 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:59:11.304707 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zjzpw_5c2d819c-e495-47ec-a1e7-4b2582a195bd/ovn-acl-logging/0.log" Jul 07 17:59:11.308358 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:59:11.308338 2583 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 17:59:11.310383 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:59:11.310363 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zjzpw_5c2d819c-e495-47ec-a1e7-4b2582a195bd/ovn-acl-logging/0.log" Jul 07 17:59:12.504243 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:59:12.504206 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:59:12.600558 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:59:12.600528 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:59:39.322709 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:59:39.322665 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:59:40.954618 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:59:40.954585 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:59:43.438227 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:59:43.438186 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:59:43.545131 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:59:43.545093 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:59:45.102091 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:59:45.102057 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:59:45.245587 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:59:45.245547 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:59:45.549362 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:59:45.549335 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:59:45.720768 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:59:45.720733 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:59:47.545082 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:59:47.545045 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:59:47.639194 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:59:47.639158 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:59:49.999268 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:59:49.999228 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:59:50.076100 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:59:50.076061 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:59:52.952536 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:59:52.952498 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:59:53.049796 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:59:53.049762 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:59:55.139266 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:59:55.139227 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:59:57.825153 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:59:57.825118 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 17:59:57.923568 ip-10-0-136-192 kubenswrapper[2583]: I0707 17:59:57.923538 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:04.972201 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:04.972114 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:05.085593 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:05.085557 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:06.895597 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:06.895560 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:18.822234 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:18.822195 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:18.924477 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:18.924441 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:19.262968 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:19.262859 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:19.380841 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:19.380801 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-58f6946598-w6cps"] Jul 07 18:00:19.384272 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:19.384254 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58f6946598-w6cps" Jul 07 18:00:19.391385 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:19.391363 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 07 18:00:19.391795 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:19.391768 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 07 18:00:19.392026 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:19.391826 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 07 18:00:19.392136 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:19.391826 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 07 18:00:19.392136 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:19.391862 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-sgjqx\"" Jul 07 18:00:19.393529 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:19.393512 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 07 18:00:19.401652 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:19.401628 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58f6946598-w6cps"] Jul 07 18:00:19.413572 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:19.413535 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:19.455938 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:19.455863 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff4c870c-40bd-43b1-91c3-1f437a5349c3-metrics-cert\") pod \"lws-controller-manager-58f6946598-w6cps\" (UID: \"ff4c870c-40bd-43b1-91c3-1f437a5349c3\") " pod="openshift-lws-operator/lws-controller-manager-58f6946598-w6cps" Jul 07 18:00:19.455938 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:19.455933 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff4c870c-40bd-43b1-91c3-1f437a5349c3-cert\") pod \"lws-controller-manager-58f6946598-w6cps\" (UID: \"ff4c870c-40bd-43b1-91c3-1f437a5349c3\") " pod="openshift-lws-operator/lws-controller-manager-58f6946598-w6cps" Jul 07 18:00:19.456184 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:19.456004 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ff4c870c-40bd-43b1-91c3-1f437a5349c3-manager-config\") pod \"lws-controller-manager-58f6946598-w6cps\" (UID: \"ff4c870c-40bd-43b1-91c3-1f437a5349c3\") " pod="openshift-lws-operator/lws-controller-manager-58f6946598-w6cps" Jul 07 18:00:19.456184 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:19.456040 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxn85\" (UniqueName: \"kubernetes.io/projected/ff4c870c-40bd-43b1-91c3-1f437a5349c3-kube-api-access-lxn85\") pod \"lws-controller-manager-58f6946598-w6cps\" (UID: \"ff4c870c-40bd-43b1-91c3-1f437a5349c3\") " pod="openshift-lws-operator/lws-controller-manager-58f6946598-w6cps" Jul 07 18:00:19.527592 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:19.527504 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:19.557125 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:19.557087 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff4c870c-40bd-43b1-91c3-1f437a5349c3-metrics-cert\") pod \"lws-controller-manager-58f6946598-w6cps\" (UID: \"ff4c870c-40bd-43b1-91c3-1f437a5349c3\") " pod="openshift-lws-operator/lws-controller-manager-58f6946598-w6cps" Jul 07 18:00:19.557318 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:19.557136 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff4c870c-40bd-43b1-91c3-1f437a5349c3-cert\") pod \"lws-controller-manager-58f6946598-w6cps\" (UID: \"ff4c870c-40bd-43b1-91c3-1f437a5349c3\") " pod="openshift-lws-operator/lws-controller-manager-58f6946598-w6cps" Jul 07 18:00:19.557318 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:19.557186 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ff4c870c-40bd-43b1-91c3-1f437a5349c3-manager-config\") pod \"lws-controller-manager-58f6946598-w6cps\" (UID: \"ff4c870c-40bd-43b1-91c3-1f437a5349c3\") " pod="openshift-lws-operator/lws-controller-manager-58f6946598-w6cps" Jul 07 18:00:19.557318 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:19.557302 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxn85\" (UniqueName: \"kubernetes.io/projected/ff4c870c-40bd-43b1-91c3-1f437a5349c3-kube-api-access-lxn85\") pod \"lws-controller-manager-58f6946598-w6cps\" (UID: \"ff4c870c-40bd-43b1-91c3-1f437a5349c3\") " pod="openshift-lws-operator/lws-controller-manager-58f6946598-w6cps" Jul 07 18:00:19.557977 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:19.557951 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ff4c870c-40bd-43b1-91c3-1f437a5349c3-manager-config\") pod \"lws-controller-manager-58f6946598-w6cps\" (UID: \"ff4c870c-40bd-43b1-91c3-1f437a5349c3\") " pod="openshift-lws-operator/lws-controller-manager-58f6946598-w6cps" Jul 07 18:00:19.560140 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:19.560119 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ff4c870c-40bd-43b1-91c3-1f437a5349c3-metrics-cert\") pod \"lws-controller-manager-58f6946598-w6cps\" (UID: \"ff4c870c-40bd-43b1-91c3-1f437a5349c3\") " pod="openshift-lws-operator/lws-controller-manager-58f6946598-w6cps" Jul 07 18:00:19.560328 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:19.560309 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff4c870c-40bd-43b1-91c3-1f437a5349c3-cert\") pod \"lws-controller-manager-58f6946598-w6cps\" (UID: \"ff4c870c-40bd-43b1-91c3-1f437a5349c3\") " pod="openshift-lws-operator/lws-controller-manager-58f6946598-w6cps" Jul 07 18:00:19.572036 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:19.572002 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxn85\" (UniqueName: \"kubernetes.io/projected/ff4c870c-40bd-43b1-91c3-1f437a5349c3-kube-api-access-lxn85\") pod \"lws-controller-manager-58f6946598-w6cps\" (UID: \"ff4c870c-40bd-43b1-91c3-1f437a5349c3\") " pod="openshift-lws-operator/lws-controller-manager-58f6946598-w6cps" Jul 07 18:00:19.693593 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:19.693555 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58f6946598-w6cps" Jul 07 18:00:19.852192 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:19.852162 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58f6946598-w6cps"] Jul 07 18:00:19.855338 ip-10-0-136-192 kubenswrapper[2583]: W0707 18:00:19.855309 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff4c870c_40bd_43b1_91c3_1f437a5349c3.slice/crio-29fe313bcf6d4b76e69d53ecbb992aaa24c44fb32cb3451df97afa6edb98b21a WatchSource:0}: Error finding container 29fe313bcf6d4b76e69d53ecbb992aaa24c44fb32cb3451df97afa6edb98b21a: Status 404 returned error can't find the container with id 29fe313bcf6d4b76e69d53ecbb992aaa24c44fb32cb3451df97afa6edb98b21a Jul 07 18:00:19.857298 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:19.857283 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 18:00:20.191039 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:20.190999 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58f6946598-w6cps" event={"ID":"ff4c870c-40bd-43b1-91c3-1f437a5349c3","Type":"ContainerStarted","Data":"29fe313bcf6d4b76e69d53ecbb992aaa24c44fb32cb3451df97afa6edb98b21a"} Jul 07 18:00:20.514795 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:20.514688 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:20.618969 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:20.618932 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:20.707320 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:20.707290 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76cd65fb98-6cfbr"] Jul 07 18:00:20.711984 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:20.711959 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-6cfbr" Jul 07 18:00:20.714533 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:20.714510 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 18:00:20.714822 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:20.714795 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 18:00:20.714933 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:20.714859 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 18:00:20.714933 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:20.714862 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-sp2sj\"" Jul 07 18:00:20.715052 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:20.714932 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 07 18:00:20.731811 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:20.731771 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76cd65fb98-6cfbr"] Jul 07 18:00:20.766614 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:20.766519 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1bb166ca-6b5b-4ed8-af7c-97d60eed5e94-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76cd65fb98-6cfbr\" (UID: \"1bb166ca-6b5b-4ed8-af7c-97d60eed5e94\") " pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-6cfbr" Jul 07 18:00:20.766614 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:20.766581 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1bb166ca-6b5b-4ed8-af7c-97d60eed5e94-manifests\") pod \"opendatahub-operator-controller-manager-76cd65fb98-6cfbr\" (UID: \"1bb166ca-6b5b-4ed8-af7c-97d60eed5e94\") " pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-6cfbr" Jul 07 18:00:20.766807 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:20.766627 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1bb166ca-6b5b-4ed8-af7c-97d60eed5e94-webhook-cert\") pod \"opendatahub-operator-controller-manager-76cd65fb98-6cfbr\" (UID: \"1bb166ca-6b5b-4ed8-af7c-97d60eed5e94\") " pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-6cfbr" Jul 07 18:00:20.766807 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:20.766649 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bb166ca-6b5b-4ed8-af7c-97d60eed5e94-tmp\") pod \"opendatahub-operator-controller-manager-76cd65fb98-6cfbr\" (UID: \"1bb166ca-6b5b-4ed8-af7c-97d60eed5e94\") " pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-6cfbr" Jul 07 18:00:20.766807 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:20.766779 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mflh\" (UniqueName: \"kubernetes.io/projected/1bb166ca-6b5b-4ed8-af7c-97d60eed5e94-kube-api-access-7mflh\") pod \"opendatahub-operator-controller-manager-76cd65fb98-6cfbr\" (UID: \"1bb166ca-6b5b-4ed8-af7c-97d60eed5e94\") " pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-6cfbr" Jul 07 18:00:20.867737 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:20.867701 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1bb166ca-6b5b-4ed8-af7c-97d60eed5e94-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76cd65fb98-6cfbr\" (UID: \"1bb166ca-6b5b-4ed8-af7c-97d60eed5e94\") " pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-6cfbr" Jul 07 18:00:20.868217 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:20.867759 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1bb166ca-6b5b-4ed8-af7c-97d60eed5e94-manifests\") pod \"opendatahub-operator-controller-manager-76cd65fb98-6cfbr\" (UID: \"1bb166ca-6b5b-4ed8-af7c-97d60eed5e94\") " pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-6cfbr" Jul 07 18:00:20.868217 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:20.867789 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1bb166ca-6b5b-4ed8-af7c-97d60eed5e94-webhook-cert\") pod \"opendatahub-operator-controller-manager-76cd65fb98-6cfbr\" (UID: \"1bb166ca-6b5b-4ed8-af7c-97d60eed5e94\") " pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-6cfbr" Jul 07 18:00:20.868217 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:20.867810 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bb166ca-6b5b-4ed8-af7c-97d60eed5e94-tmp\") pod \"opendatahub-operator-controller-manager-76cd65fb98-6cfbr\" (UID: \"1bb166ca-6b5b-4ed8-af7c-97d60eed5e94\") " pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-6cfbr" Jul 07 18:00:20.868217 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:20.867885 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mflh\" (UniqueName: \"kubernetes.io/projected/1bb166ca-6b5b-4ed8-af7c-97d60eed5e94-kube-api-access-7mflh\") pod \"opendatahub-operator-controller-manager-76cd65fb98-6cfbr\" (UID: \"1bb166ca-6b5b-4ed8-af7c-97d60eed5e94\") " pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-6cfbr" Jul 07 18:00:20.868390 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:20.868228 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1bb166ca-6b5b-4ed8-af7c-97d60eed5e94-manifests\") pod \"opendatahub-operator-controller-manager-76cd65fb98-6cfbr\" (UID: \"1bb166ca-6b5b-4ed8-af7c-97d60eed5e94\") " pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-6cfbr" Jul 07 18:00:20.870466 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:20.870440 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1bb166ca-6b5b-4ed8-af7c-97d60eed5e94-tmp\") pod \"opendatahub-operator-controller-manager-76cd65fb98-6cfbr\" (UID: \"1bb166ca-6b5b-4ed8-af7c-97d60eed5e94\") " pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-6cfbr" Jul 07 18:00:20.870692 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:20.870670 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1bb166ca-6b5b-4ed8-af7c-97d60eed5e94-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76cd65fb98-6cfbr\" (UID: \"1bb166ca-6b5b-4ed8-af7c-97d60eed5e94\") " pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-6cfbr" Jul 07 18:00:20.870734 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:20.870689 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1bb166ca-6b5b-4ed8-af7c-97d60eed5e94-webhook-cert\") pod \"opendatahub-operator-controller-manager-76cd65fb98-6cfbr\" (UID: \"1bb166ca-6b5b-4ed8-af7c-97d60eed5e94\") " pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-6cfbr" Jul 07 18:00:20.878358 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:20.878316 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:20.879589 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:20.879566 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mflh\" (UniqueName: \"kubernetes.io/projected/1bb166ca-6b5b-4ed8-af7c-97d60eed5e94-kube-api-access-7mflh\") pod \"opendatahub-operator-controller-manager-76cd65fb98-6cfbr\" (UID: \"1bb166ca-6b5b-4ed8-af7c-97d60eed5e94\") " pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-6cfbr" Jul 07 18:00:21.022810 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:21.022719 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-6cfbr" Jul 07 18:00:21.023241 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:21.023198 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:21.132608 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:21.132557 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:21.200318 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:21.200295 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76cd65fb98-6cfbr"] Jul 07 18:00:21.203572 ip-10-0-136-192 kubenswrapper[2583]: W0707 18:00:21.203535 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb166ca_6b5b_4ed8_af7c_97d60eed5e94.slice/crio-2b070b2c42777aa453ccfdd4526a12d29746f758067dd95c8987c2aca57be7bc WatchSource:0}: Error finding container 2b070b2c42777aa453ccfdd4526a12d29746f758067dd95c8987c2aca57be7bc: Status 404 returned error can't find the container with id 2b070b2c42777aa453ccfdd4526a12d29746f758067dd95c8987c2aca57be7bc Jul 07 18:00:21.239021 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:21.238973 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:21.344109 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:21.344079 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:22.208007 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:22.207941 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-6cfbr" event={"ID":"1bb166ca-6b5b-4ed8-af7c-97d60eed5e94","Type":"ContainerStarted","Data":"2b070b2c42777aa453ccfdd4526a12d29746f758067dd95c8987c2aca57be7bc"} Jul 07 18:00:25.228470 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:25.228354 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-6cfbr" event={"ID":"1bb166ca-6b5b-4ed8-af7c-97d60eed5e94","Type":"ContainerStarted","Data":"c1251b5af3c6ebbde5651c330dd22a8e25d8783d45aca7735433c42e58ef7b7b"} Jul 07 18:00:25.229831 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:25.229805 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58f6946598-w6cps" event={"ID":"ff4c870c-40bd-43b1-91c3-1f437a5349c3","Type":"ContainerStarted","Data":"545564f3e2160b3079eeafde0d30b60e5bd9f64443564797e119df73bd99607b"} Jul 07 18:00:25.229996 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:25.229921 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-58f6946598-w6cps" Jul 07 18:00:25.277930 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:25.277857 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-58f6946598-w6cps" podStartSLOduration=1.322243751 podStartE2EDuration="6.277838398s" totalImagesPullingTime="4.955594647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:00:25.277332111 +0000 UTC m=+374.359935815" watchObservedRunningTime="2026-07-07 18:00:25.277838398 +0000 UTC m=+374.360442101" Jul 07 18:00:26.236033 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:26.235995 2583 generic.go:356] "Generic (PLEG): container finished" podID="1bb166ca-6b5b-4ed8-af7c-97d60eed5e94" containerID="c1251b5af3c6ebbde5651c330dd22a8e25d8783d45aca7735433c42e58ef7b7b" exitCode=0 Jul 07 18:00:26.236501 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:26.236092 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-6cfbr" event={"ID":"1bb166ca-6b5b-4ed8-af7c-97d60eed5e94","Type":"ContainerDied","Data":"c1251b5af3c6ebbde5651c330dd22a8e25d8783d45aca7735433c42e58ef7b7b"} Jul 07 18:00:27.243054 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:27.243020 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-6cfbr" event={"ID":"1bb166ca-6b5b-4ed8-af7c-97d60eed5e94","Type":"ContainerStarted","Data":"4c20aa6f2c97b64dddf8950dd0bcec766cc375575a130814dad2d70db27525de"} Jul 07 18:00:27.243463 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:27.243159 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-6cfbr" Jul 07 18:00:27.275951 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:27.275870 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-6cfbr" podStartSLOduration=2.188789344 podStartE2EDuration="7.275854368s" totalImagesPullingTime="3.998254353s" totalInitContainerRuntime="1.088810671s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:00:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:00:27.273294947 +0000 UTC m=+376.355898653" watchObservedRunningTime="2026-07-07 18:00:27.275854368 +0000 UTC m=+376.358458072" Jul 07 18:00:36.238566 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:36.238532 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-58f6946598-w6cps" Jul 07 18:00:38.249869 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:38.249838 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-76cd65fb98-6cfbr" Jul 07 18:00:41.651946 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:41.651887 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:41.949173 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:41.949087 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:42.431172 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:42.431137 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:42.781842 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:42.781740 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:43.038112 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:43.038015 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:43.301195 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:43.301159 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:44.686025 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:44.685990 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:45.744635 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:45.744593 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:45.954722 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:45.954686 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:46.195076 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:46.195032 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:48.902856 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:48.902815 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:51.273782 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:51.273749 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:51.570836 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:51.570786 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:52.021577 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.021490 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt"] Jul 07 18:00:52.031273 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.031243 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:52.035725 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.035700 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 18:00:52.035837 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.035752 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-79mfq\"" Jul 07 18:00:52.051175 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.051137 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt"] Jul 07 18:00:52.056330 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.056303 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4r59rt\" (UID: \"6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:52.056487 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.056343 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4r59rt\" (UID: \"6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:52.056487 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.056361 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4r59rt\" (UID: \"6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:52.056487 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.056418 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4r59rt\" (UID: \"6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:52.056487 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.056445 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4r59rt\" (UID: \"6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:52.056487 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.056465 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4r59rt\" (UID: \"6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:52.056690 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.056518 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4r59rt\" (UID: \"6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:52.056690 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.056540 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgw5f\" (UniqueName: \"kubernetes.io/projected/6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2-kube-api-access-kgw5f\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4r59rt\" (UID: \"6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:52.056690 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.056559 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4r59rt\" (UID: \"6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:52.100372 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.100330 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:52.156995 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.156952 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4r59rt\" (UID: \"6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:52.156995 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.157004 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4r59rt\" (UID: \"6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:52.157226 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.157121 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4r59rt\" (UID: \"6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:52.157226 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.157173 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4r59rt\" (UID: \"6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:52.157226 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.157204 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgw5f\" (UniqueName: \"kubernetes.io/projected/6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2-kube-api-access-kgw5f\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4r59rt\" (UID: \"6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:52.157372 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.157230 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4r59rt\" (UID: \"6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:52.157372 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.157339 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4r59rt\" (UID: \"6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:52.157471 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.157375 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4r59rt\" (UID: \"6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:52.157471 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.157391 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4r59rt\" (UID: \"6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:52.157471 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.157399 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4r59rt\" (UID: \"6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:52.157471 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.157432 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4r59rt\" (UID: \"6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:52.157806 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.157784 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4r59rt\" (UID: \"6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:52.157874 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.157805 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4r59rt\" (UID: \"6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:52.158325 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.158301 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4r59rt\" (UID: \"6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:52.159707 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.159681 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4r59rt\" (UID: \"6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:52.159816 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.159795 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4r59rt\" (UID: \"6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:52.186759 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.186730 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4r59rt\" (UID: \"6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:52.186975 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.186952 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgw5f\" (UniqueName: \"kubernetes.io/projected/6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2-kube-api-access-kgw5f\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4r59rt\" (UID: \"6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:52.342005 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.341967 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:52.442977 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.442946 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:52.520002 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.519974 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt"] Jul 07 18:00:52.522686 ip-10-0-136-192 kubenswrapper[2583]: W0707 18:00:52.522648 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f1a1c19_af97_46c4_baf9_0aef3ae8d7f2.slice/crio-e06126e721041236dfec546baf560c461f73f3a74f409587432812b34b564c0f WatchSource:0}: Error finding container e06126e721041236dfec546baf560c461f73f3a74f409587432812b34b564c0f: Status 404 returned error can't find the container with id e06126e721041236dfec546baf560c461f73f3a74f409587432812b34b564c0f Jul 07 18:00:52.745309 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:52.745220 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:53.363154 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:53.363121 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" event={"ID":"6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2","Type":"ContainerStarted","Data":"e06126e721041236dfec546baf560c461f73f3a74f409587432812b34b564c0f"} Jul 07 18:00:54.232115 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:54.232081 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j4f4p"] Jul 07 18:00:54.234625 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:54.234608 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j4f4p" Jul 07 18:00:54.238735 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:54.238706 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 07 18:00:54.238893 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:54.238785 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 07 18:00:54.239609 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:54.239577 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 07 18:00:54.239609 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:54.239597 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gbnr2\"" Jul 07 18:00:54.268615 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:54.268584 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j4f4p"] Jul 07 18:00:54.274111 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:54.274083 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/27c52354-e3bb-47b5-94ca-6ec736c41d3f-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-j4f4p\" (UID: \"27c52354-e3bb-47b5-94ca-6ec736c41d3f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j4f4p" Jul 07 18:00:54.274270 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:54.274115 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/27c52354-e3bb-47b5-94ca-6ec736c41d3f-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-j4f4p\" (UID: \"27c52354-e3bb-47b5-94ca-6ec736c41d3f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j4f4p" Jul 07 18:00:54.274270 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:54.274160 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk4mc\" (UniqueName: \"kubernetes.io/projected/27c52354-e3bb-47b5-94ca-6ec736c41d3f-kube-api-access-zk4mc\") pod \"istiod-openshift-gateway-94bb8fbfd-j4f4p\" (UID: \"27c52354-e3bb-47b5-94ca-6ec736c41d3f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j4f4p" Jul 07 18:00:54.274270 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:54.274236 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/27c52354-e3bb-47b5-94ca-6ec736c41d3f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-j4f4p\" (UID: \"27c52354-e3bb-47b5-94ca-6ec736c41d3f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j4f4p" Jul 07 18:00:54.274441 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:54.274297 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/27c52354-e3bb-47b5-94ca-6ec736c41d3f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-j4f4p\" (UID: \"27c52354-e3bb-47b5-94ca-6ec736c41d3f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j4f4p" Jul 07 18:00:54.274441 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:54.274359 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27c52354-e3bb-47b5-94ca-6ec736c41d3f-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-j4f4p\" (UID: \"27c52354-e3bb-47b5-94ca-6ec736c41d3f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j4f4p" Jul 07 18:00:54.274521 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:54.274443 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/27c52354-e3bb-47b5-94ca-6ec736c41d3f-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-j4f4p\" (UID: \"27c52354-e3bb-47b5-94ca-6ec736c41d3f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j4f4p" Jul 07 18:00:54.375018 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:54.374977 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/27c52354-e3bb-47b5-94ca-6ec736c41d3f-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-j4f4p\" (UID: \"27c52354-e3bb-47b5-94ca-6ec736c41d3f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j4f4p" Jul 07 18:00:54.375497 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:54.375025 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/27c52354-e3bb-47b5-94ca-6ec736c41d3f-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-j4f4p\" (UID: \"27c52354-e3bb-47b5-94ca-6ec736c41d3f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j4f4p" Jul 07 18:00:54.375497 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:54.375092 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk4mc\" (UniqueName: \"kubernetes.io/projected/27c52354-e3bb-47b5-94ca-6ec736c41d3f-kube-api-access-zk4mc\") pod \"istiod-openshift-gateway-94bb8fbfd-j4f4p\" (UID: \"27c52354-e3bb-47b5-94ca-6ec736c41d3f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j4f4p" Jul 07 18:00:54.375497 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:54.375126 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/27c52354-e3bb-47b5-94ca-6ec736c41d3f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-j4f4p\" (UID: \"27c52354-e3bb-47b5-94ca-6ec736c41d3f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j4f4p" Jul 07 18:00:54.375497 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:54.375174 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/27c52354-e3bb-47b5-94ca-6ec736c41d3f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-j4f4p\" (UID: \"27c52354-e3bb-47b5-94ca-6ec736c41d3f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j4f4p" Jul 07 18:00:54.375497 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:54.375234 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27c52354-e3bb-47b5-94ca-6ec736c41d3f-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-j4f4p\" (UID: \"27c52354-e3bb-47b5-94ca-6ec736c41d3f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j4f4p" Jul 07 18:00:54.375497 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:54.375272 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/27c52354-e3bb-47b5-94ca-6ec736c41d3f-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-j4f4p\" (UID: \"27c52354-e3bb-47b5-94ca-6ec736c41d3f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j4f4p" Jul 07 18:00:54.376346 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:54.376292 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/27c52354-e3bb-47b5-94ca-6ec736c41d3f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-j4f4p\" (UID: \"27c52354-e3bb-47b5-94ca-6ec736c41d3f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j4f4p" Jul 07 18:00:54.377875 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:54.377848 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/27c52354-e3bb-47b5-94ca-6ec736c41d3f-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-j4f4p\" (UID: \"27c52354-e3bb-47b5-94ca-6ec736c41d3f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j4f4p" Jul 07 18:00:54.378089 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:54.378064 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/27c52354-e3bb-47b5-94ca-6ec736c41d3f-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-j4f4p\" (UID: \"27c52354-e3bb-47b5-94ca-6ec736c41d3f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j4f4p" Jul 07 18:00:54.378181 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:54.378136 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/27c52354-e3bb-47b5-94ca-6ec736c41d3f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-j4f4p\" (UID: \"27c52354-e3bb-47b5-94ca-6ec736c41d3f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j4f4p" Jul 07 18:00:54.378181 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:54.378171 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/27c52354-e3bb-47b5-94ca-6ec736c41d3f-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-j4f4p\" (UID: \"27c52354-e3bb-47b5-94ca-6ec736c41d3f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j4f4p" Jul 07 18:00:54.392018 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:54.391977 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27c52354-e3bb-47b5-94ca-6ec736c41d3f-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-j4f4p\" (UID: \"27c52354-e3bb-47b5-94ca-6ec736c41d3f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j4f4p" Jul 07 18:00:54.393321 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:54.393298 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk4mc\" (UniqueName: \"kubernetes.io/projected/27c52354-e3bb-47b5-94ca-6ec736c41d3f-kube-api-access-zk4mc\") pod \"istiod-openshift-gateway-94bb8fbfd-j4f4p\" (UID: \"27c52354-e3bb-47b5-94ca-6ec736c41d3f\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j4f4p" Jul 07 18:00:54.545332 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:54.545299 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j4f4p" Jul 07 18:00:54.774889 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:54.774778 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j4f4p"] Jul 07 18:00:54.777609 ip-10-0-136-192 kubenswrapper[2583]: W0707 18:00:54.777575 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c52354_e3bb_47b5_94ca_6ec736c41d3f.slice/crio-56af4db28f65938e2f0f637cbeeb772401d0b407263c61239bf8dae7c2d2f585 WatchSource:0}: Error finding container 56af4db28f65938e2f0f637cbeeb772401d0b407263c61239bf8dae7c2d2f585: Status 404 returned error can't find the container with id 56af4db28f65938e2f0f637cbeeb772401d0b407263c61239bf8dae7c2d2f585 Jul 07 18:00:55.375474 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:55.375433 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j4f4p" event={"ID":"27c52354-e3bb-47b5-94ca-6ec736c41d3f","Type":"ContainerStarted","Data":"56af4db28f65938e2f0f637cbeeb772401d0b407263c61239bf8dae7c2d2f585"} Jul 07 18:00:56.261028 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:56.260988 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:00:56.261164 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:56.261076 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:00:56.261164 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:56.261145 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:00:56.382681 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:56.382643 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" event={"ID":"6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2","Type":"ContainerStarted","Data":"911f00da49bedffe6c99711606595a84121f2a3fe3dca2a86aead5bb195ed6b6"} Jul 07 18:00:56.425878 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:56.424219 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" podStartSLOduration=1.686801271 podStartE2EDuration="5.422622419s" totalImagesPullingTime="3.735821148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:00:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:00:56.416389357 +0000 UTC m=+405.498993188" watchObservedRunningTime="2026-07-07 18:00:56.422622419 +0000 UTC m=+405.505226123" Jul 07 18:00:57.342198 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:57.342163 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:57.348266 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:57.348239 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:57.387290 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:57.387258 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:57.388207 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:57.388186 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4r59rt" Jul 07 18:00:58.414092 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:58.414006 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:58.707217 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:58.707119 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:58.997958 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:58.997832 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:59.172209 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:59.172170 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:00:59.172347 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:59.172251 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:00:59.275934 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:59.275812 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:00:59.399820 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:59.399783 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j4f4p" event={"ID":"27c52354-e3bb-47b5-94ca-6ec736c41d3f","Type":"ContainerStarted","Data":"3a14d9ad7ef6eadecf2ff025935d067ee913f5b59018dbb0481e3036c145952e"} Jul 07 18:00:59.400049 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:59.400033 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j4f4p" Jul 07 18:00:59.710019 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:00:59.709974 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:00.406720 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:01:00.406691 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j4f4p" Jul 07 18:01:00.484631 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:01:00.484573 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j4f4p" podStartSLOduration=2.092335848 podStartE2EDuration="6.484547161s" totalImagesPullingTime="4.392211313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:00:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:00:59.630452658 +0000 UTC m=+408.713056361" watchObservedRunningTime="2026-07-07 18:01:00.484547161 +0000 UTC m=+409.567150868" Jul 07 18:01:43.052498 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:01:43.052404 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:43.231796 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:01:43.231757 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:43.391453 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:01:43.391355 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:50.547188 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:01:50.547151 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:50.683081 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:01:50.683051 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:52.198692 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:01:52.198669 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:01:52.311722 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:01:52.311688 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:11.898110 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:11.898064 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:12.492216 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:12.492180 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:13.631532 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:13.631494 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:13.790348 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:13.790309 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:16.973490 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:16.973441 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:17.448084 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:17.448049 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:17.577592 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:17.577554 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:17.688623 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:17.688561 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:18.135100 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:18.135056 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:18.265392 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:18.265355 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:19.793402 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:19.793316 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:19.983504 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:19.983465 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:22.295156 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:22.295112 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:22.491069 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:22.491036 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:24.836289 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:24.836255 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:27.302157 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:27.302116 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:27.401664 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:27.401628 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:27.510623 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:27.510587 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:27.611003 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:27.610915 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:28.353558 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:28.353521 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:28.454764 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:28.454732 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:29.646483 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:29.646445 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:31.155771 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:31.155731 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:31.305966 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:31.305927 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:35.186364 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:35.186328 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:35.382077 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:35.382043 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:36.082761 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:36.082724 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:36.595177 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:36.595145 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:38.917407 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:38.917372 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:39.023571 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:39.023535 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:47.334488 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:47.334454 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:47.869570 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:47.869535 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:48.250041 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:48.249955 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:49.832339 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:49.832305 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:50.029972 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:50.029893 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:51.516920 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:51.516879 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:51.747089 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:51.747054 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:55.883113 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:55.883081 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:56.100186 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:56.100137 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:59.476801 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:59.476764 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:02:59.658198 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:02:59.658168 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:03:25.385296 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:25.385252 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:03:25.452037 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:25.452005 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:03:36.090231 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:36.090187 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:03:36.388136 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:36.388022 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:03:37.720196 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:37.720165 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:03:37.784968 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:37.784927 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:03:42.768787 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:42.768746 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:03:43.478169 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:43.478139 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:03:43.563229 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:43.563157 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:03:43.664137 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:43.664105 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:03:43.801142 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:43.801107 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:03:43.803787 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:43.803765 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-jfmtn" Jul 07 18:03:43.806230 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:43.806212 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 18:03:43.806444 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:43.806432 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xxlhd\"" Jul 07 18:03:43.806501 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:43.806482 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 18:03:43.807014 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:43.806997 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 07 18:03:43.816246 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:43.816198 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:03:43.855854 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:43.855816 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:03:43.962171 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:43.962136 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4f756f4c-3846-4888-bc28-70c2dc1f9bfb-config-file\") pod \"limitador-limitador-54d6d988d5-jfmtn\" (UID: \"4f756f4c-3846-4888-bc28-70c2dc1f9bfb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jfmtn" Jul 07 18:03:43.962171 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:43.962182 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcw9d\" (UniqueName: \"kubernetes.io/projected/4f756f4c-3846-4888-bc28-70c2dc1f9bfb-kube-api-access-kcw9d\") pod \"limitador-limitador-54d6d988d5-jfmtn\" (UID: \"4f756f4c-3846-4888-bc28-70c2dc1f9bfb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jfmtn" Jul 07 18:03:44.063875 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:44.063788 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4f756f4c-3846-4888-bc28-70c2dc1f9bfb-config-file\") pod \"limitador-limitador-54d6d988d5-jfmtn\" (UID: \"4f756f4c-3846-4888-bc28-70c2dc1f9bfb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jfmtn" Jul 07 18:03:44.063875 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:44.063824 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcw9d\" (UniqueName: \"kubernetes.io/projected/4f756f4c-3846-4888-bc28-70c2dc1f9bfb-kube-api-access-kcw9d\") pod \"limitador-limitador-54d6d988d5-jfmtn\" (UID: \"4f756f4c-3846-4888-bc28-70c2dc1f9bfb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jfmtn" Jul 07 18:03:44.064520 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:44.064499 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4f756f4c-3846-4888-bc28-70c2dc1f9bfb-config-file\") pod \"limitador-limitador-54d6d988d5-jfmtn\" (UID: \"4f756f4c-3846-4888-bc28-70c2dc1f9bfb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jfmtn" Jul 07 18:03:44.073144 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:44.073104 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcw9d\" (UniqueName: \"kubernetes.io/projected/4f756f4c-3846-4888-bc28-70c2dc1f9bfb-kube-api-access-kcw9d\") pod \"limitador-limitador-54d6d988d5-jfmtn\" (UID: \"4f756f4c-3846-4888-bc28-70c2dc1f9bfb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jfmtn" Jul 07 18:03:44.123996 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:44.123964 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-jfmtn" Jul 07 18:03:44.261508 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:44.261483 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:03:44.264284 ip-10-0-136-192 kubenswrapper[2583]: W0707 18:03:44.264251 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f756f4c_3846_4888_bc28_70c2dc1f9bfb.slice/crio-12abd596fbcf0aa156b9ad6b4716fd726ebc87bdc9e41521112120e2f8d3e585 WatchSource:0}: Error finding container 12abd596fbcf0aa156b9ad6b4716fd726ebc87bdc9e41521112120e2f8d3e585: Status 404 returned error can't find the container with id 12abd596fbcf0aa156b9ad6b4716fd726ebc87bdc9e41521112120e2f8d3e585 Jul 07 18:03:44.311329 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:44.311292 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:03:44.385360 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:44.385263 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:03:45.189411 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:45.189370 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-jfmtn" event={"ID":"4f756f4c-3846-4888-bc28-70c2dc1f9bfb","Type":"ContainerStarted","Data":"12abd596fbcf0aa156b9ad6b4716fd726ebc87bdc9e41521112120e2f8d3e585"} Jul 07 18:03:47.201113 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:47.201070 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-jfmtn" event={"ID":"4f756f4c-3846-4888-bc28-70c2dc1f9bfb","Type":"ContainerStarted","Data":"4c3f4f0a294a69663b1c52cac09a831f19a2ce8e41c033b8f1a12b5e3d7b7c6d"} Jul 07 18:03:47.201571 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:47.201215 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-jfmtn" Jul 07 18:03:47.224404 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:47.224340 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-jfmtn" podStartSLOduration=1.84474831 podStartE2EDuration="4.224322658s" totalImagesPullingTime="2.379574348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:03:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:03:47.221521278 +0000 UTC m=+576.304124984" watchObservedRunningTime="2026-07-07 18:03:47.224322658 +0000 UTC m=+576.306926363" Jul 07 18:03:58.206964 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:03:58.206925 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-jfmtn" Jul 07 18:04:00.040590 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:04:00.040547 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:04:11.330178 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:04:11.330143 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wqfm4_997df85c-7632-47b5-a8a6-1c37f3346251/console-operator/2.log" Jul 07 18:04:11.332375 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:04:11.332352 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wqfm4_997df85c-7632-47b5-a8a6-1c37f3346251/console-operator/2.log" Jul 07 18:04:11.337566 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:04:11.337545 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zjzpw_5c2d819c-e495-47ec-a1e7-4b2582a195bd/ovn-acl-logging/0.log" Jul 07 18:04:11.345826 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:04:11.345782 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zjzpw_5c2d819c-e495-47ec-a1e7-4b2582a195bd/ovn-acl-logging/0.log" Jul 07 18:04:12.883352 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:04:12.883312 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:04:13.488209 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:04:13.488178 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:04:14.778584 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:04:14.778551 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:04:20.226711 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:04:20.226669 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:04:20.409086 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:04:20.409050 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:04:20.500631 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:04:20.500537 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:04:23.554059 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:04:23.554020 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:04:23.647421 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:04:23.647385 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:04:57.342712 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:04:57.342671 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:04:58.657382 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:04:58.657344 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:04:59.247511 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:04:59.247454 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:05:09.022626 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:05:09.022564 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:05:09.123247 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:05:09.123207 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:05:18.018833 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:05:18.018790 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:05:18.551585 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:05:18.551540 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:05:24.902461 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:05:24.902427 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:05:25.648058 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:05:25.648025 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:05:29.825525 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:05:29.825486 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:05:43.043569 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:05:43.043537 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:05:43.140700 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:05:43.140658 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:05:44.242648 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:05:44.242613 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:06:00.467955 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:06:00.467887 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:06:00.964040 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:06:00.964000 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:06:02.268491 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:06:02.268404 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:06:08.480060 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:06:08.480024 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:06:10.550541 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:06:10.550502 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:06:23.424394 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:06:23.424356 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:06:25.752057 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:06:25.752020 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:07:13.654006 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:07:13.653970 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:07:13.913080 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:07:13.912988 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:07:14.903951 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:07:14.903877 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:07:15.003750 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:07:15.003703 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:07:43.150831 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:07:43.150745 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:07:52.466477 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:07:52.466384 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:08:05.300303 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:08:05.300269 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:08:12.747756 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:08:12.747719 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:08:23.357605 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:08:23.357572 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:08:26.508563 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:08:26.508525 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:08:27.821319 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:08:27.821280 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:08:51.561118 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:08:51.561085 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:09:03.648199 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:09:03.648108 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:09:11.201869 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:09:11.201833 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:09:11.362944 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:09:11.362880 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:09:11.369076 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:09:11.369046 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wqfm4_997df85c-7632-47b5-a8a6-1c37f3346251/console-operator/2.log" Jul 07 18:09:11.369729 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:09:11.369712 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wqfm4_997df85c-7632-47b5-a8a6-1c37f3346251/console-operator/2.log" Jul 07 18:09:11.376047 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:09:11.376022 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zjzpw_5c2d819c-e495-47ec-a1e7-4b2582a195bd/ovn-acl-logging/0.log" Jul 07 18:09:11.376846 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:09:11.376812 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zjzpw_5c2d819c-e495-47ec-a1e7-4b2582a195bd/ovn-acl-logging/0.log" Jul 07 18:09:11.478327 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:09:11.478236 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:09:18.573270 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:09:18.573225 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:09:18.718335 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:09:18.718297 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:09:18.907278 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:09:18.907176 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:09:19.077784 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:09:19.077750 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:09:25.459068 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:09:25.459027 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:09:25.561169 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:09:25.561128 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:09:42.547525 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:09:42.547486 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:09:51.859357 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:09:51.859317 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:10:00.144506 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:10:00.144468 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:10:10.735668 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:10:10.735626 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:10:20.046461 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:10:20.046424 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:10:29.545238 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:10:29.545202 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:10:41.642611 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:10:41.642530 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:10:51.141763 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:10:51.141726 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:11:13.744391 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:11:13.744359 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:11:25.744135 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:11:25.744093 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:11:49.144935 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:11:49.144881 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:11:54.629301 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:11:54.629266 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:11:55.945207 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:11:55.945166 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:11:57.135310 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:11:57.135273 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:11:58.939742 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:11:58.939702 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:12:14.541876 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:12:14.541792 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:12:23.553292 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:12:23.553249 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:12:32.550261 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:12:32.550218 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:12:43.554914 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:12:43.554859 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:12:52.537634 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:12:52.537592 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:13:11.282487 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:13:11.282453 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:13:11.342620 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:13:11.342590 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:14:11.399981 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:14:11.399949 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wqfm4_997df85c-7632-47b5-a8a6-1c37f3346251/console-operator/2.log" Jul 07 18:14:11.400894 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:14:11.400873 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wqfm4_997df85c-7632-47b5-a8a6-1c37f3346251/console-operator/2.log" Jul 07 18:14:11.406702 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:14:11.406679 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zjzpw_5c2d819c-e495-47ec-a1e7-4b2582a195bd/ovn-acl-logging/0.log" Jul 07 18:14:11.407733 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:14:11.407705 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zjzpw_5c2d819c-e495-47ec-a1e7-4b2582a195bd/ovn-acl-logging/0.log" Jul 07 18:14:40.152475 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:14:40.152436 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:14:42.543404 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:14:42.543367 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:14:54.150390 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:14:54.150352 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:15:03.852219 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:15:03.852129 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:15:15.253703 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:15:15.253660 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:15:26.836229 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:15:26.836193 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:15:37.335182 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:15:37.335142 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:15:45.646744 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:15:45.646711 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:15:56.643284 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:15:56.643244 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:16:07.968734 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:16:07.968695 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:16:17.565918 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:16:17.565870 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:16:27.854179 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:16:27.854143 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:16:52.794733 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:16:52.794636 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:16:59.873286 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:16:59.873250 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:17:01.789419 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:17:01.789375 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:17:09.185018 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:17:09.184981 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:17:20.048274 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:17:20.048234 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:17:37.476507 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:17:37.476472 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:17:44.223612 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:17:44.223580 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:17:44.322503 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:17:44.322470 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:17:48.453537 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:17:48.453498 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:17:56.751765 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:17:56.751726 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:18:08.248456 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:18:08.248356 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:18:17.248494 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:18:17.248458 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:18:26.133305 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:18:26.133260 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:18:39.139481 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:18:39.139435 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:18:49.638751 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:18:49.638711 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:18:58.846651 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:18:58.846607 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:19:11.142674 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:19:11.142633 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:19:11.431649 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:19:11.431569 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wqfm4_997df85c-7632-47b5-a8a6-1c37f3346251/console-operator/2.log" Jul 07 18:19:11.432199 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:19:11.432176 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wqfm4_997df85c-7632-47b5-a8a6-1c37f3346251/console-operator/2.log" Jul 07 18:19:11.438680 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:19:11.438651 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zjzpw_5c2d819c-e495-47ec-a1e7-4b2582a195bd/ovn-acl-logging/0.log" Jul 07 18:19:11.438838 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:19:11.438826 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zjzpw_5c2d819c-e495-47ec-a1e7-4b2582a195bd/ovn-acl-logging/0.log" Jul 07 18:19:22.338077 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:19:22.338035 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:19:31.159507 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:19:31.159459 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:19:42.447872 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:19:42.447830 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:19:53.853162 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:19:53.853122 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:20:06.244528 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:06.244488 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:20:09.107597 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:09.107548 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:13.802274 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:13.802238 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:18.939531 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:18.939498 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:20:22.136668 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:22.136637 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:27.237111 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:27.237074 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:30.218135 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:30.218098 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:30.337591 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:30.337556 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:30.694095 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:30.694064 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:31.271856 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:31.271812 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:20:37.148037 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.147989 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:37.241824 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.241793 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:37.370926 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.370861 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz"] Jul 07 18:20:37.375259 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.375230 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:37.377849 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.377821 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2e64e8e2-openshift-default-dockercfg-dxnfs\"" Jul 07 18:20:37.401781 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.401691 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz"] Jul 07 18:20:37.407055 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.407024 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:37.429571 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.429531 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a93996e9-ade0-4c37-8698-3adb81e8458a-istiod-ca-cert\") pod \"e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:37.429754 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.429695 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a93996e9-ade0-4c37-8698-3adb81e8458a-istio-podinfo\") pod \"e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:37.429754 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.429734 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a93996e9-ade0-4c37-8698-3adb81e8458a-istio-token\") pod \"e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:37.429858 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.429777 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a93996e9-ade0-4c37-8698-3adb81e8458a-istio-envoy\") pod \"e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:37.429858 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.429816 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a93996e9-ade0-4c37-8698-3adb81e8458a-workload-certs\") pod \"e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:37.429954 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.429882 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93996e9-ade0-4c37-8698-3adb81e8458a-credential-socket\") pod \"e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:37.429954 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.429945 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxzp9\" (UniqueName: \"kubernetes.io/projected/a93996e9-ade0-4c37-8698-3adb81e8458a-kube-api-access-kxzp9\") pod \"e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:37.430055 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.430037 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93996e9-ade0-4c37-8698-3adb81e8458a-workload-socket\") pod \"e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:37.430099 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.430076 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a93996e9-ade0-4c37-8698-3adb81e8458a-istio-data\") pod \"e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:37.505173 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.505134 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:37.531370 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.531334 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a93996e9-ade0-4c37-8698-3adb81e8458a-istio-podinfo\") pod \"e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:37.531516 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.531375 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a93996e9-ade0-4c37-8698-3adb81e8458a-istio-token\") pod \"e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:37.531516 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.531406 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a93996e9-ade0-4c37-8698-3adb81e8458a-istio-envoy\") pod \"e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:37.531516 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.531439 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a93996e9-ade0-4c37-8698-3adb81e8458a-workload-certs\") pod \"e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:37.531516 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.531467 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93996e9-ade0-4c37-8698-3adb81e8458a-credential-socket\") pod \"e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:37.531516 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.531495 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxzp9\" (UniqueName: \"kubernetes.io/projected/a93996e9-ade0-4c37-8698-3adb81e8458a-kube-api-access-kxzp9\") pod \"e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:37.531759 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.531637 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93996e9-ade0-4c37-8698-3adb81e8458a-workload-socket\") pod \"e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:37.531759 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.531714 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a93996e9-ade0-4c37-8698-3adb81e8458a-istio-data\") pod \"e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:37.531865 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.531759 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a93996e9-ade0-4c37-8698-3adb81e8458a-istiod-ca-cert\") pod \"e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:37.531979 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.531956 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a93996e9-ade0-4c37-8698-3adb81e8458a-workload-certs\") pod \"e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:37.532106 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.532082 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93996e9-ade0-4c37-8698-3adb81e8458a-credential-socket\") pod \"e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:37.532187 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.532141 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a93996e9-ade0-4c37-8698-3adb81e8458a-istio-data\") pod \"e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:37.532258 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.532180 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93996e9-ade0-4c37-8698-3adb81e8458a-workload-socket\") pod \"e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:37.532484 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.532434 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a93996e9-ade0-4c37-8698-3adb81e8458a-istiod-ca-cert\") pod \"e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:37.533993 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.533969 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a93996e9-ade0-4c37-8698-3adb81e8458a-istio-envoy\") pod \"e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:37.534150 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.534132 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a93996e9-ade0-4c37-8698-3adb81e8458a-istio-podinfo\") pod \"e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:37.540515 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.540487 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a93996e9-ade0-4c37-8698-3adb81e8458a-istio-token\") pod \"e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:37.540721 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.540698 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxzp9\" (UniqueName: \"kubernetes.io/projected/a93996e9-ade0-4c37-8698-3adb81e8458a-kube-api-access-kxzp9\") pod \"e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:37.686693 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.686594 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:37.890816 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.890787 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz"] Jul 07 18:20:37.893502 ip-10-0-136-192 kubenswrapper[2583]: W0707 18:20:37.893464 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda93996e9_ade0_4c37_8698_3adb81e8458a.slice/crio-be7182ae022e81b7489fc6353242ec45127fef7e5880a2fee4f00bb4b589fb4a WatchSource:0}: Error finding container be7182ae022e81b7489fc6353242ec45127fef7e5880a2fee4f00bb4b589fb4a: Status 404 returned error can't find the container with id be7182ae022e81b7489fc6353242ec45127fef7e5880a2fee4f00bb4b589fb4a Jul 07 18:20:37.895757 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.895722 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:20:37.895832 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.895798 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:20:37.895832 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:37.895827 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:20:38.033659 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:38.033623 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" event={"ID":"a93996e9-ade0-4c37-8698-3adb81e8458a","Type":"ContainerStarted","Data":"eaee3a9448ba9309b354572a18449749c33384909d0eb9365199f8991338fd9f"} Jul 07 18:20:38.033846 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:38.033685 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" event={"ID":"a93996e9-ade0-4c37-8698-3adb81e8458a","Type":"ContainerStarted","Data":"be7182ae022e81b7489fc6353242ec45127fef7e5880a2fee4f00bb4b589fb4a"} Jul 07 18:20:38.063318 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:38.063263 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" podStartSLOduration=1.063244019 podStartE2EDuration="1.063244019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:20:38.061760788 +0000 UTC m=+1587.144364490" watchObservedRunningTime="2026-07-07 18:20:38.063244019 +0000 UTC m=+1587.145847722" Jul 07 18:20:38.587853 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:38.587810 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:38.682271 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:38.682238 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:38.687226 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:38.687203 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:38.692176 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:38.692147 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:38.802839 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:38.802804 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv"] Jul 07 18:20:38.806550 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:38.806531 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv" Jul 07 18:20:38.808833 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:38.808807 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-2e64e8e2-serving-cert\"" Jul 07 18:20:38.809003 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:38.808940 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-jnv2d\"" Jul 07 18:20:38.822932 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:38.822876 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv"] Jul 07 18:20:38.834302 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:38.834267 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:38.846000 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:38.845888 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4d8n\" (UniqueName: \"kubernetes.io/projected/03121dc7-36b0-4a6a-a367-27fc687792b2-kube-api-access-b4d8n\") pod \"maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv\" (UID: \"03121dc7-36b0-4a6a-a367-27fc687792b2\") " pod="opendatahub/maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv" Jul 07 18:20:38.846000 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:38.845945 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03121dc7-36b0-4a6a-a367-27fc687792b2-maas-api-tls\") pod \"maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv\" (UID: \"03121dc7-36b0-4a6a-a367-27fc687792b2\") " pod="opendatahub/maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv" Jul 07 18:20:38.846000 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:38.845988 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03121dc7-36b0-4a6a-a367-27fc687792b2-openshift-service-ca\") pod \"maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv\" (UID: \"03121dc7-36b0-4a6a-a367-27fc687792b2\") " pod="opendatahub/maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv" Jul 07 18:20:38.936381 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:38.936345 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:38.946916 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:38.946869 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4d8n\" (UniqueName: \"kubernetes.io/projected/03121dc7-36b0-4a6a-a367-27fc687792b2-kube-api-access-b4d8n\") pod \"maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv\" (UID: \"03121dc7-36b0-4a6a-a367-27fc687792b2\") " pod="opendatahub/maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv" Jul 07 18:20:38.947059 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:38.946954 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03121dc7-36b0-4a6a-a367-27fc687792b2-maas-api-tls\") pod \"maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv\" (UID: \"03121dc7-36b0-4a6a-a367-27fc687792b2\") " pod="opendatahub/maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv" Jul 07 18:20:38.947059 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:38.947003 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03121dc7-36b0-4a6a-a367-27fc687792b2-openshift-service-ca\") pod \"maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv\" (UID: \"03121dc7-36b0-4a6a-a367-27fc687792b2\") " pod="opendatahub/maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv" Jul 07 18:20:38.947670 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:38.947637 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03121dc7-36b0-4a6a-a367-27fc687792b2-openshift-service-ca\") pod \"maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv\" (UID: \"03121dc7-36b0-4a6a-a367-27fc687792b2\") " pod="opendatahub/maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv" Jul 07 18:20:38.949686 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:38.949662 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03121dc7-36b0-4a6a-a367-27fc687792b2-maas-api-tls\") pod \"maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv\" (UID: \"03121dc7-36b0-4a6a-a367-27fc687792b2\") " pod="opendatahub/maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv" Jul 07 18:20:38.956698 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:38.956656 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4d8n\" (UniqueName: \"kubernetes.io/projected/03121dc7-36b0-4a6a-a367-27fc687792b2-kube-api-access-b4d8n\") pod \"maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv\" (UID: \"03121dc7-36b0-4a6a-a367-27fc687792b2\") " pod="opendatahub/maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv" Jul 07 18:20:39.039358 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:39.039318 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:39.040375 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:39.040353 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:39.118362 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:39.118276 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv" Jul 07 18:20:39.282472 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:39.282435 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv"] Jul 07 18:20:39.284052 ip-10-0-136-192 kubenswrapper[2583]: W0707 18:20:39.284018 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03121dc7_36b0_4a6a_a367_27fc687792b2.slice/crio-1a1382c8ce63c6840e392c2fd2312559e36c7e4b794ce3671ff01509eab59f92 WatchSource:0}: Error finding container 1a1382c8ce63c6840e392c2fd2312559e36c7e4b794ce3671ff01509eab59f92: Status 404 returned error can't find the container with id 1a1382c8ce63c6840e392c2fd2312559e36c7e4b794ce3671ff01509eab59f92 Jul 07 18:20:39.285965 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:39.285895 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 18:20:39.426509 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:39.426393 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:39.589195 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:39.589154 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:40.049271 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:40.049232 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv" event={"ID":"03121dc7-36b0-4a6a-a367-27fc687792b2","Type":"ContainerStarted","Data":"1a1382c8ce63c6840e392c2fd2312559e36c7e4b794ce3671ff01509eab59f92"} Jul 07 18:20:41.488982 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:41.488939 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:42.062427 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:42.062391 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv" event={"ID":"03121dc7-36b0-4a6a-a367-27fc687792b2","Type":"ContainerStarted","Data":"c5eddc1e2e26e8790ffac4f3e17139a12d2e84c9a1f55997628e3bfee0f14f47"} Jul 07 18:20:42.062596 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:42.062472 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv" Jul 07 18:20:42.087857 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:42.087800 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv" podStartSLOduration=1.882278326 podStartE2EDuration="4.087784325s" totalImagesPullingTime="2.205505999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:20:42.086303398 +0000 UTC m=+1591.168907103" watchObservedRunningTime="2026-07-07 18:20:42.087784325 +0000 UTC m=+1591.170388028" Jul 07 18:20:43.636736 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:43.636703 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:43.809929 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:43.809879 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz"] Jul 07 18:20:43.810156 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:43.810132 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" podUID="a93996e9-ade0-4c37-8698-3adb81e8458a" containerName="istio-proxy" containerID="cri-o://eaee3a9448ba9309b354572a18449749c33384909d0eb9365199f8991338fd9f" gracePeriod=30 Jul 07 18:20:43.827260 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:43.827224 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:43.928063 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:43.927979 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:43.963544 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:43.963506 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv"] Jul 07 18:20:44.029127 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:44.029098 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:44.073308 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:44.073258 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv" podUID="03121dc7-36b0-4a6a-a367-27fc687792b2" containerName="maas-api" containerID="cri-o://c5eddc1e2e26e8790ffac4f3e17139a12d2e84c9a1f55997628e3bfee0f14f47" gracePeriod=30 Jul 07 18:20:44.330428 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:44.330399 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv" Jul 07 18:20:44.401778 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:44.401741 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03121dc7-36b0-4a6a-a367-27fc687792b2-maas-api-tls\") pod \"03121dc7-36b0-4a6a-a367-27fc687792b2\" (UID: \"03121dc7-36b0-4a6a-a367-27fc687792b2\") " Jul 07 18:20:44.401778 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:44.401781 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03121dc7-36b0-4a6a-a367-27fc687792b2-openshift-service-ca\") pod \"03121dc7-36b0-4a6a-a367-27fc687792b2\" (UID: \"03121dc7-36b0-4a6a-a367-27fc687792b2\") " Jul 07 18:20:44.402091 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:44.401885 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4d8n\" (UniqueName: \"kubernetes.io/projected/03121dc7-36b0-4a6a-a367-27fc687792b2-kube-api-access-b4d8n\") pod \"03121dc7-36b0-4a6a-a367-27fc687792b2\" (UID: \"03121dc7-36b0-4a6a-a367-27fc687792b2\") " Jul 07 18:20:44.402202 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:44.402177 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03121dc7-36b0-4a6a-a367-27fc687792b2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "03121dc7-36b0-4a6a-a367-27fc687792b2" (UID: "03121dc7-36b0-4a6a-a367-27fc687792b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:20:44.404244 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:44.404210 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03121dc7-36b0-4a6a-a367-27fc687792b2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "03121dc7-36b0-4a6a-a367-27fc687792b2" (UID: "03121dc7-36b0-4a6a-a367-27fc687792b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:20:44.404244 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:44.404207 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03121dc7-36b0-4a6a-a367-27fc687792b2-kube-api-access-b4d8n" (OuterVolumeSpecName: "kube-api-access-b4d8n") pod "03121dc7-36b0-4a6a-a367-27fc687792b2" (UID: "03121dc7-36b0-4a6a-a367-27fc687792b2"). InnerVolumeSpecName "kube-api-access-b4d8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:20:44.503417 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:44.503314 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4d8n\" (UniqueName: \"kubernetes.io/projected/03121dc7-36b0-4a6a-a367-27fc687792b2-kube-api-access-b4d8n\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:20:44.503417 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:44.503355 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03121dc7-36b0-4a6a-a367-27fc687792b2-maas-api-tls\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:20:44.503417 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:44.503370 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03121dc7-36b0-4a6a-a367-27fc687792b2-openshift-service-ca\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:20:45.079571 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:45.079536 2583 generic.go:356] "Generic (PLEG): container finished" podID="03121dc7-36b0-4a6a-a367-27fc687792b2" containerID="c5eddc1e2e26e8790ffac4f3e17139a12d2e84c9a1f55997628e3bfee0f14f47" exitCode=0 Jul 07 18:20:45.080050 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:45.079579 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv" Jul 07 18:20:45.080050 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:45.079578 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv" event={"ID":"03121dc7-36b0-4a6a-a367-27fc687792b2","Type":"ContainerDied","Data":"c5eddc1e2e26e8790ffac4f3e17139a12d2e84c9a1f55997628e3bfee0f14f47"} Jul 07 18:20:45.080050 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:45.079705 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv" event={"ID":"03121dc7-36b0-4a6a-a367-27fc687792b2","Type":"ContainerDied","Data":"1a1382c8ce63c6840e392c2fd2312559e36c7e4b794ce3671ff01509eab59f92"} Jul 07 18:20:45.080050 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:45.079736 2583 scope.go:117] "RemoveContainer" containerID="c5eddc1e2e26e8790ffac4f3e17139a12d2e84c9a1f55997628e3bfee0f14f47" Jul 07 18:20:45.089429 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:45.089411 2583 scope.go:117] "RemoveContainer" containerID="c5eddc1e2e26e8790ffac4f3e17139a12d2e84c9a1f55997628e3bfee0f14f47" Jul 07 18:20:45.089744 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:20:45.089722 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5eddc1e2e26e8790ffac4f3e17139a12d2e84c9a1f55997628e3bfee0f14f47\": container with ID starting with c5eddc1e2e26e8790ffac4f3e17139a12d2e84c9a1f55997628e3bfee0f14f47 not found: ID does not exist" containerID="c5eddc1e2e26e8790ffac4f3e17139a12d2e84c9a1f55997628e3bfee0f14f47" Jul 07 18:20:45.089819 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:45.089757 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5eddc1e2e26e8790ffac4f3e17139a12d2e84c9a1f55997628e3bfee0f14f47"} err="failed to get container status \"c5eddc1e2e26e8790ffac4f3e17139a12d2e84c9a1f55997628e3bfee0f14f47\": rpc error: code = NotFound desc = could not find container \"c5eddc1e2e26e8790ffac4f3e17139a12d2e84c9a1f55997628e3bfee0f14f47\": container with ID starting with c5eddc1e2e26e8790ffac4f3e17139a12d2e84c9a1f55997628e3bfee0f14f47 not found: ID does not exist" Jul 07 18:20:45.102636 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:45.102603 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv"] Jul 07 18:20:45.110089 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:45.110061 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-2e64e8e2-658f7f75d5-gxrrv"] Jul 07 18:20:45.206364 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:45.206332 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:45.429554 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:45.429458 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03121dc7-36b0-4a6a-a367-27fc687792b2" path="/var/lib/kubelet/pods/03121dc7-36b0-4a6a-a367-27fc687792b2/volumes" Jul 07 18:20:49.065022 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.064994 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:49.101294 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.101202 2583 generic.go:356] "Generic (PLEG): container finished" podID="a93996e9-ade0-4c37-8698-3adb81e8458a" containerID="eaee3a9448ba9309b354572a18449749c33384909d0eb9365199f8991338fd9f" exitCode=0 Jul 07 18:20:49.101436 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.101294 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" event={"ID":"a93996e9-ade0-4c37-8698-3adb81e8458a","Type":"ContainerDied","Data":"eaee3a9448ba9309b354572a18449749c33384909d0eb9365199f8991338fd9f"} Jul 07 18:20:49.101436 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.101302 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" Jul 07 18:20:49.101436 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.101383 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz" event={"ID":"a93996e9-ade0-4c37-8698-3adb81e8458a","Type":"ContainerDied","Data":"be7182ae022e81b7489fc6353242ec45127fef7e5880a2fee4f00bb4b589fb4a"} Jul 07 18:20:49.101436 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.101416 2583 scope.go:117] "RemoveContainer" containerID="eaee3a9448ba9309b354572a18449749c33384909d0eb9365199f8991338fd9f" Jul 07 18:20:49.110671 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.110650 2583 scope.go:117] "RemoveContainer" containerID="eaee3a9448ba9309b354572a18449749c33384909d0eb9365199f8991338fd9f" Jul 07 18:20:49.110970 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:20:49.110946 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaee3a9448ba9309b354572a18449749c33384909d0eb9365199f8991338fd9f\": container with ID starting with eaee3a9448ba9309b354572a18449749c33384909d0eb9365199f8991338fd9f not found: ID does not exist" containerID="eaee3a9448ba9309b354572a18449749c33384909d0eb9365199f8991338fd9f" Jul 07 18:20:49.111056 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.110984 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaee3a9448ba9309b354572a18449749c33384909d0eb9365199f8991338fd9f"} err="failed to get container status \"eaee3a9448ba9309b354572a18449749c33384909d0eb9365199f8991338fd9f\": rpc error: code = NotFound desc = could not find container \"eaee3a9448ba9309b354572a18449749c33384909d0eb9365199f8991338fd9f\": container with ID starting with eaee3a9448ba9309b354572a18449749c33384909d0eb9365199f8991338fd9f not found: ID does not exist" Jul 07 18:20:49.151170 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.151131 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a93996e9-ade0-4c37-8698-3adb81e8458a-istio-envoy\") pod \"a93996e9-ade0-4c37-8698-3adb81e8458a\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " Jul 07 18:20:49.151365 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.151184 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a93996e9-ade0-4c37-8698-3adb81e8458a-istio-token\") pod \"a93996e9-ade0-4c37-8698-3adb81e8458a\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " Jul 07 18:20:49.151365 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.151203 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a93996e9-ade0-4c37-8698-3adb81e8458a-istio-data\") pod \"a93996e9-ade0-4c37-8698-3adb81e8458a\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " Jul 07 18:20:49.151365 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.151222 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a93996e9-ade0-4c37-8698-3adb81e8458a-workload-certs\") pod \"a93996e9-ade0-4c37-8698-3adb81e8458a\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " Jul 07 18:20:49.151537 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.151380 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93996e9-ade0-4c37-8698-3adb81e8458a-credential-socket\") pod \"a93996e9-ade0-4c37-8698-3adb81e8458a\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " Jul 07 18:20:49.151537 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.151460 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93996e9-ade0-4c37-8698-3adb81e8458a-workload-socket\") pod \"a93996e9-ade0-4c37-8698-3adb81e8458a\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " Jul 07 18:20:49.151537 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.151496 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93996e9-ade0-4c37-8698-3adb81e8458a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a93996e9-ade0-4c37-8698-3adb81e8458a" (UID: "a93996e9-ade0-4c37-8698-3adb81e8458a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:20:49.151537 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.151511 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxzp9\" (UniqueName: \"kubernetes.io/projected/a93996e9-ade0-4c37-8698-3adb81e8458a-kube-api-access-kxzp9\") pod \"a93996e9-ade0-4c37-8698-3adb81e8458a\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " Jul 07 18:20:49.151537 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.151527 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93996e9-ade0-4c37-8698-3adb81e8458a-istio-data" (OuterVolumeSpecName: "istio-data") pod "a93996e9-ade0-4c37-8698-3adb81e8458a" (UID: "a93996e9-ade0-4c37-8698-3adb81e8458a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:20:49.151796 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.151541 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a93996e9-ade0-4c37-8698-3adb81e8458a-istio-podinfo\") pod \"a93996e9-ade0-4c37-8698-3adb81e8458a\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " Jul 07 18:20:49.151796 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.151614 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a93996e9-ade0-4c37-8698-3adb81e8458a-istiod-ca-cert\") pod \"a93996e9-ade0-4c37-8698-3adb81e8458a\" (UID: \"a93996e9-ade0-4c37-8698-3adb81e8458a\") " Jul 07 18:20:49.151796 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.151685 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93996e9-ade0-4c37-8698-3adb81e8458a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a93996e9-ade0-4c37-8698-3adb81e8458a" (UID: "a93996e9-ade0-4c37-8698-3adb81e8458a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:20:49.151796 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.151727 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93996e9-ade0-4c37-8698-3adb81e8458a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a93996e9-ade0-4c37-8698-3adb81e8458a" (UID: "a93996e9-ade0-4c37-8698-3adb81e8458a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:20:49.152040 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.151975 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93996e9-ade0-4c37-8698-3adb81e8458a-credential-socket\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:20:49.152040 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.151995 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93996e9-ade0-4c37-8698-3adb81e8458a-workload-socket\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:20:49.152040 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.152011 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a93996e9-ade0-4c37-8698-3adb81e8458a-istio-data\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:20:49.152040 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.152024 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a93996e9-ade0-4c37-8698-3adb81e8458a-workload-certs\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:20:49.152402 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.152069 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a93996e9-ade0-4c37-8698-3adb81e8458a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a93996e9-ade0-4c37-8698-3adb81e8458a" (UID: "a93996e9-ade0-4c37-8698-3adb81e8458a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:20:49.153883 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.153854 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a93996e9-ade0-4c37-8698-3adb81e8458a-kube-api-access-kxzp9" (OuterVolumeSpecName: "kube-api-access-kxzp9") pod "a93996e9-ade0-4c37-8698-3adb81e8458a" (UID: "a93996e9-ade0-4c37-8698-3adb81e8458a"). InnerVolumeSpecName "kube-api-access-kxzp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:20:49.154026 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.153883 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93996e9-ade0-4c37-8698-3adb81e8458a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a93996e9-ade0-4c37-8698-3adb81e8458a" (UID: "a93996e9-ade0-4c37-8698-3adb81e8458a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:20:49.154188 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.154164 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a93996e9-ade0-4c37-8698-3adb81e8458a-istio-token" (OuterVolumeSpecName: "istio-token") pod "a93996e9-ade0-4c37-8698-3adb81e8458a" (UID: "a93996e9-ade0-4c37-8698-3adb81e8458a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:20:49.154289 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.154194 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a93996e9-ade0-4c37-8698-3adb81e8458a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a93996e9-ade0-4c37-8698-3adb81e8458a" (UID: "a93996e9-ade0-4c37-8698-3adb81e8458a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 18:20:49.253088 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.253047 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxzp9\" (UniqueName: \"kubernetes.io/projected/a93996e9-ade0-4c37-8698-3adb81e8458a-kube-api-access-kxzp9\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:20:49.253088 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.253081 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a93996e9-ade0-4c37-8698-3adb81e8458a-istio-podinfo\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:20:49.253088 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.253092 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a93996e9-ade0-4c37-8698-3adb81e8458a-istiod-ca-cert\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:20:49.253088 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.253102 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a93996e9-ade0-4c37-8698-3adb81e8458a-istio-envoy\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:20:49.253397 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.253110 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a93996e9-ade0-4c37-8698-3adb81e8458a-istio-token\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:20:49.429758 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.429727 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz"] Jul 07 18:20:49.438247 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:49.438218 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2e64e8e2-openshift-default-945b548c7-xs9cz"] Jul 07 18:20:50.390188 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.390152 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:50.483555 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.483522 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:50.605486 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.605447 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78"] Jul 07 18:20:50.606077 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.606056 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a93996e9-ade0-4c37-8698-3adb81e8458a" containerName="istio-proxy" Jul 07 18:20:50.606127 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.606083 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a93996e9-ade0-4c37-8698-3adb81e8458a" containerName="istio-proxy" Jul 07 18:20:50.606127 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.606111 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03121dc7-36b0-4a6a-a367-27fc687792b2" containerName="maas-api" Jul 07 18:20:50.606127 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.606120 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="03121dc7-36b0-4a6a-a367-27fc687792b2" containerName="maas-api" Jul 07 18:20:50.606222 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.606214 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a93996e9-ade0-4c37-8698-3adb81e8458a" containerName="istio-proxy" Jul 07 18:20:50.606259 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.606228 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="03121dc7-36b0-4a6a-a367-27fc687792b2" containerName="maas-api" Jul 07 18:20:50.609643 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.609620 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:50.614336 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.614312 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d7f738e5-openshift-default-dockercfg-6x62j\"" Jul 07 18:20:50.638351 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.638314 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78"] Jul 07 18:20:50.665172 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.665068 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-credential-socket\") pod \"e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:50.665341 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.665188 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-workload-certs\") pod \"e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:50.665341 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.665219 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-istio-envoy\") pod \"e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:50.665341 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.665242 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-istio-token\") pod \"e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:50.665341 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.665271 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-istio-data\") pod \"e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:50.665341 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.665302 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4mkq\" (UniqueName: \"kubernetes.io/projected/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-kube-api-access-v4mkq\") pod \"e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:50.665341 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.665326 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-istiod-ca-cert\") pod \"e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:50.665581 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.665351 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-istio-podinfo\") pod \"e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:50.665581 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.665456 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-workload-socket\") pod \"e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:50.711042 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.710983 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:50.766499 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.766459 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-workload-certs\") pod \"e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:50.766499 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.766497 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-istio-envoy\") pod \"e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:50.766749 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.766515 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-istio-token\") pod \"e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:50.766749 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.766542 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-istio-data\") pod \"e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:50.766749 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.766656 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4mkq\" (UniqueName: \"kubernetes.io/projected/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-kube-api-access-v4mkq\") pod \"e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:50.766749 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.766699 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-istiod-ca-cert\") pod \"e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:50.766749 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.766739 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-istio-podinfo\") pod \"e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:50.767058 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.766833 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-workload-socket\") pod \"e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:50.767058 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.766873 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-credential-socket\") pod \"e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:50.767058 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.766887 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-workload-certs\") pod \"e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:50.767058 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.766939 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-istio-data\") pod \"e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:50.767212 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.767166 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-workload-socket\") pod \"e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:50.767246 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.767232 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-credential-socket\") pod \"e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:50.767431 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.767397 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-istiod-ca-cert\") pod \"e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:50.769063 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.769043 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-istio-envoy\") pod \"e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:50.769323 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.769303 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-istio-podinfo\") pod \"e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:50.783470 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.783440 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4mkq\" (UniqueName: \"kubernetes.io/projected/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-kube-api-access-v4mkq\") pod \"e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:50.783470 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.783456 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-istio-token\") pod \"e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:50.876890 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.876853 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:50.920617 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:50.920521 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:51.078216 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:51.078191 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78"] Jul 07 18:20:51.080292 ip-10-0-136-192 kubenswrapper[2583]: W0707 18:20:51.080250 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82c7ec7c_f1db_4287_8ffa_9b3cd581f991.slice/crio-1158e1388a978037a3c570250f48763e49ef96b0bb55f78663679287468b780d WatchSource:0}: Error finding container 1158e1388a978037a3c570250f48763e49ef96b0bb55f78663679287468b780d: Status 404 returned error can't find the container with id 1158e1388a978037a3c570250f48763e49ef96b0bb55f78663679287468b780d Jul 07 18:20:51.082416 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:51.082385 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:20:51.082489 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:51.082454 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:20:51.082489 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:51.082483 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:20:51.113979 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:51.113948 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" event={"ID":"82c7ec7c-f1db-4287-8ffa-9b3cd581f991","Type":"ContainerStarted","Data":"1158e1388a978037a3c570250f48763e49ef96b0bb55f78663679287468b780d"} Jul 07 18:20:51.429885 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:51.429848 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a93996e9-ade0-4c37-8698-3adb81e8458a" path="/var/lib/kubelet/pods/a93996e9-ade0-4c37-8698-3adb81e8458a/volumes" Jul 07 18:20:51.857050 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:51.857020 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:51.954014 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:51.953979 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:51.981450 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:51.981410 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm"] Jul 07 18:20:51.984095 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:51.984072 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm" Jul 07 18:20:51.986612 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:51.986592 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-jnv2d\"" Jul 07 18:20:51.986721 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:51.986592 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-d7f738e5-serving-cert\"" Jul 07 18:20:52.000321 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:52.000287 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm"] Jul 07 18:20:52.081235 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:52.081196 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbrw4\" (UniqueName: \"kubernetes.io/projected/4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a-kube-api-access-bbrw4\") pod \"maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm\" (UID: \"4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a\") " pod="opendatahub/maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm" Jul 07 18:20:52.081418 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:52.081317 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a-maas-api-tls\") pod \"maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm\" (UID: \"4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a\") " pod="opendatahub/maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm" Jul 07 18:20:52.081418 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:52.081403 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a-openshift-service-ca\") pod \"maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm\" (UID: \"4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a\") " pod="opendatahub/maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm" Jul 07 18:20:52.120743 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:52.120646 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" event={"ID":"82c7ec7c-f1db-4287-8ffa-9b3cd581f991","Type":"ContainerStarted","Data":"4eae82dd4dd49019ca1d1602a8d171685a5850d3955c9d167b971fdf3337348d"} Jul 07 18:20:52.182777 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:52.182729 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a-openshift-service-ca\") pod \"maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm\" (UID: \"4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a\") " pod="opendatahub/maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm" Jul 07 18:20:52.183023 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:52.182973 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbrw4\" (UniqueName: \"kubernetes.io/projected/4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a-kube-api-access-bbrw4\") pod \"maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm\" (UID: \"4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a\") " pod="opendatahub/maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm" Jul 07 18:20:52.183128 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:52.183107 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a-maas-api-tls\") pod \"maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm\" (UID: \"4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a\") " pod="opendatahub/maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm" Jul 07 18:20:52.183198 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:20:52.183178 2583 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-d7f738e5-serving-cert: secret "maas-api-e2e-ait-d7f738e5-serving-cert" not found Jul 07 18:20:52.183258 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:20:52.183247 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a-maas-api-tls podName:4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a nodeName:}" failed. No retries permitted until 2026-07-07 18:20:52.683226419 +0000 UTC m=+1601.765830113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a-maas-api-tls") pod "maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm" (UID: "4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a") : secret "maas-api-e2e-ait-d7f738e5-serving-cert" not found Jul 07 18:20:52.183547 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:52.183527 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a-openshift-service-ca\") pod \"maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm\" (UID: \"4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a\") " pod="opendatahub/maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm" Jul 07 18:20:52.203226 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:52.203173 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" podStartSLOduration=2.203155197 podStartE2EDuration="2.203155197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:20:52.200661769 +0000 UTC m=+1601.283265470" watchObservedRunningTime="2026-07-07 18:20:52.203155197 +0000 UTC m=+1601.285758878" Jul 07 18:20:52.209091 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:52.209055 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:52.238795 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:52.238757 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbrw4\" (UniqueName: \"kubernetes.io/projected/4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a-kube-api-access-bbrw4\") pod \"maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm\" (UID: \"4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a\") " pod="opendatahub/maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm" Jul 07 18:20:52.418739 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:52.418617 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:52.688318 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:52.688204 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a-maas-api-tls\") pod \"maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm\" (UID: \"4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a\") " pod="opendatahub/maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm" Jul 07 18:20:52.690814 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:52.690791 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a-maas-api-tls\") pod \"maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm\" (UID: \"4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a\") " pod="opendatahub/maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm" Jul 07 18:20:52.889256 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:52.889206 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:52.895049 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:52.895022 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm" Jul 07 18:20:52.921206 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:52.921174 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:52.927006 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:52.926979 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:52.963926 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:52.963876 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:53.044005 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:53.043931 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm"] Jul 07 18:20:53.047081 ip-10-0-136-192 kubenswrapper[2583]: W0707 18:20:53.047051 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4149f3b8_7bd2_4d44_9d0b_7ba7b7d2472a.slice/crio-a94f69fc44a12cd92dfb758b687eab439117ad64583e88d391303e9ae60f60b8 WatchSource:0}: Error finding container a94f69fc44a12cd92dfb758b687eab439117ad64583e88d391303e9ae60f60b8: Status 404 returned error can't find the container with id a94f69fc44a12cd92dfb758b687eab439117ad64583e88d391303e9ae60f60b8 Jul 07 18:20:53.127649 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:53.127617 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm" event={"ID":"4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a","Type":"ContainerStarted","Data":"a94f69fc44a12cd92dfb758b687eab439117ad64583e88d391303e9ae60f60b8"} Jul 07 18:20:53.127871 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:53.127859 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:53.128945 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:53.128925 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:20:53.926350 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:53.926315 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:54.135036 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:54.134931 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm" event={"ID":"4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a","Type":"ContainerStarted","Data":"96b9b7e61d14839c45ab6ce17c7492c2b3d82a404c5586dfbbf7710d9727f055"} Jul 07 18:20:54.135036 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:54.135010 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm" Jul 07 18:20:54.156996 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:54.156946 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm" podStartSLOduration=2.719459188 podStartE2EDuration="3.156929608s" totalImagesPullingTime="437.47042ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:20:54.156515064 +0000 UTC m=+1603.239118768" watchObservedRunningTime="2026-07-07 18:20:54.156929608 +0000 UTC m=+1603.239533310" Jul 07 18:20:57.004787 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:57.004753 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:57.504730 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:57.504695 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm"] Jul 07 18:20:57.504981 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:57.504937 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm" podUID="4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a" containerName="maas-api" containerID="cri-o://96b9b7e61d14839c45ab6ce17c7492c2b3d82a404c5586dfbbf7710d9727f055" gracePeriod=30 Jul 07 18:20:57.542089 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:57.542055 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:20:57.775124 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:57.775097 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm" Jul 07 18:20:57.850158 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:57.850119 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a-openshift-service-ca\") pod \"4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a\" (UID: \"4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a\") " Jul 07 18:20:57.850337 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:57.850187 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbrw4\" (UniqueName: \"kubernetes.io/projected/4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a-kube-api-access-bbrw4\") pod \"4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a\" (UID: \"4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a\") " Jul 07 18:20:57.850337 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:57.850248 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a-maas-api-tls\") pod \"4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a\" (UID: \"4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a\") " Jul 07 18:20:57.850499 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:57.850475 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a" (UID: "4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:20:57.852461 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:57.852438 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a-kube-api-access-bbrw4" (OuterVolumeSpecName: "kube-api-access-bbrw4") pod "4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a" (UID: "4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a"). InnerVolumeSpecName "kube-api-access-bbrw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:20:57.852543 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:57.852513 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a" (UID: "4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:20:57.951318 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:57.951275 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a-openshift-service-ca\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:20:57.951318 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:57.951311 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbrw4\" (UniqueName: \"kubernetes.io/projected/4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a-kube-api-access-bbrw4\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:20:57.951318 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:57.951322 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a-maas-api-tls\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:20:58.159167 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:58.159128 2583 generic.go:356] "Generic (PLEG): container finished" podID="4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a" containerID="96b9b7e61d14839c45ab6ce17c7492c2b3d82a404c5586dfbbf7710d9727f055" exitCode=0 Jul 07 18:20:58.159729 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:58.159181 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm" Jul 07 18:20:58.159729 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:58.159215 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm" event={"ID":"4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a","Type":"ContainerDied","Data":"96b9b7e61d14839c45ab6ce17c7492c2b3d82a404c5586dfbbf7710d9727f055"} Jul 07 18:20:58.159729 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:58.159271 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm" event={"ID":"4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a","Type":"ContainerDied","Data":"a94f69fc44a12cd92dfb758b687eab439117ad64583e88d391303e9ae60f60b8"} Jul 07 18:20:58.159729 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:58.159294 2583 scope.go:117] "RemoveContainer" containerID="96b9b7e61d14839c45ab6ce17c7492c2b3d82a404c5586dfbbf7710d9727f055" Jul 07 18:20:58.168881 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:58.168863 2583 scope.go:117] "RemoveContainer" containerID="96b9b7e61d14839c45ab6ce17c7492c2b3d82a404c5586dfbbf7710d9727f055" Jul 07 18:20:58.169202 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:20:58.169178 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96b9b7e61d14839c45ab6ce17c7492c2b3d82a404c5586dfbbf7710d9727f055\": container with ID starting with 96b9b7e61d14839c45ab6ce17c7492c2b3d82a404c5586dfbbf7710d9727f055 not found: ID does not exist" containerID="96b9b7e61d14839c45ab6ce17c7492c2b3d82a404c5586dfbbf7710d9727f055" Jul 07 18:20:58.169251 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:58.169213 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b9b7e61d14839c45ab6ce17c7492c2b3d82a404c5586dfbbf7710d9727f055"} err="failed to get container status \"96b9b7e61d14839c45ab6ce17c7492c2b3d82a404c5586dfbbf7710d9727f055\": rpc error: code = NotFound desc = could not find container \"96b9b7e61d14839c45ab6ce17c7492c2b3d82a404c5586dfbbf7710d9727f055\": container with ID starting with 96b9b7e61d14839c45ab6ce17c7492c2b3d82a404c5586dfbbf7710d9727f055 not found: ID does not exist" Jul 07 18:20:58.200184 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:58.200149 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm"] Jul 07 18:20:58.202234 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:58.202205 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-d7f738e5-69bbc678c5-9lsrm"] Jul 07 18:20:59.429663 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:20:59.429632 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a" path="/var/lib/kubelet/pods/4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a/volumes" Jul 07 18:21:02.865774 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:02.865695 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:21:03.216651 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:03.216538 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78"] Jul 07 18:21:03.216879 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:03.216843 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" podUID="82c7ec7c-f1db-4287-8ffa-9b3cd581f991" containerName="istio-proxy" containerID="cri-o://4eae82dd4dd49019ca1d1602a8d171685a5850d3955c9d167b971fdf3337348d" gracePeriod=30 Jul 07 18:21:03.296716 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:03.296668 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:21:03.459194 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:03.459158 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:21:05.064839 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:05.064807 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:21:08.130547 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:08.130505 2583 patch_prober.go:28] interesting pod/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 18:21:08.131003 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:08.130616 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" podUID="82c7ec7c-f1db-4287-8ffa-9b3cd581f991" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 18:21:08.477940 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:08.477890 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:21:08.657834 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:08.657746 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-istio-token\") pod \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " Jul 07 18:21:08.657834 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:08.657804 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-credential-socket\") pod \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " Jul 07 18:21:08.657834 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:08.657822 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-workload-certs\") pod \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " Jul 07 18:21:08.658127 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:08.657852 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-istio-podinfo\") pod \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " Jul 07 18:21:08.658127 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:08.657885 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4mkq\" (UniqueName: \"kubernetes.io/projected/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-kube-api-access-v4mkq\") pod \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " Jul 07 18:21:08.658127 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:08.657954 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-workload-socket\") pod \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " Jul 07 18:21:08.658127 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:08.658109 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "82c7ec7c-f1db-4287-8ffa-9b3cd581f991" (UID: "82c7ec7c-f1db-4287-8ffa-9b3cd581f991"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:21:08.658302 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:08.658159 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-istio-data\") pod \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " Jul 07 18:21:08.658302 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:08.658208 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "82c7ec7c-f1db-4287-8ffa-9b3cd581f991" (UID: "82c7ec7c-f1db-4287-8ffa-9b3cd581f991"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:21:08.658302 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:08.658266 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "82c7ec7c-f1db-4287-8ffa-9b3cd581f991" (UID: "82c7ec7c-f1db-4287-8ffa-9b3cd581f991"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:21:08.658439 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:08.658397 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-istio-data" (OuterVolumeSpecName: "istio-data") pod "82c7ec7c-f1db-4287-8ffa-9b3cd581f991" (UID: "82c7ec7c-f1db-4287-8ffa-9b3cd581f991"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:21:08.658495 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:08.658468 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-istio-envoy\") pod \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " Jul 07 18:21:08.658547 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:08.658512 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-istiod-ca-cert\") pod \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\" (UID: \"82c7ec7c-f1db-4287-8ffa-9b3cd581f991\") " Jul 07 18:21:08.658950 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:08.658921 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "82c7ec7c-f1db-4287-8ffa-9b3cd581f991" (UID: "82c7ec7c-f1db-4287-8ffa-9b3cd581f991"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:21:08.659134 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:08.659113 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-workload-socket\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:21:08.659211 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:08.659141 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-istio-data\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:21:08.659211 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:08.659157 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-istiod-ca-cert\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:21:08.659211 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:08.659170 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-credential-socket\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:21:08.659211 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:08.659181 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-workload-certs\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:21:08.660302 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:08.660264 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-istio-token" (OuterVolumeSpecName: "istio-token") pod "82c7ec7c-f1db-4287-8ffa-9b3cd581f991" (UID: "82c7ec7c-f1db-4287-8ffa-9b3cd581f991"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:21:08.660462 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:08.660371 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-kube-api-access-v4mkq" (OuterVolumeSpecName: "kube-api-access-v4mkq") pod "82c7ec7c-f1db-4287-8ffa-9b3cd581f991" (UID: "82c7ec7c-f1db-4287-8ffa-9b3cd581f991"). InnerVolumeSpecName "kube-api-access-v4mkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:21:08.660841 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:08.660824 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "82c7ec7c-f1db-4287-8ffa-9b3cd581f991" (UID: "82c7ec7c-f1db-4287-8ffa-9b3cd581f991"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:21:08.661042 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:08.661021 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "82c7ec7c-f1db-4287-8ffa-9b3cd581f991" (UID: "82c7ec7c-f1db-4287-8ffa-9b3cd581f991"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 18:21:08.760213 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:08.760178 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-istio-envoy\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:21:08.760213 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:08.760209 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-istio-token\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:21:08.760213 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:08.760218 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-istio-podinfo\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:21:08.760442 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:08.760226 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4mkq\" (UniqueName: \"kubernetes.io/projected/82c7ec7c-f1db-4287-8ffa-9b3cd581f991-kube-api-access-v4mkq\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:21:09.215708 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:09.215669 2583 generic.go:356] "Generic (PLEG): container finished" podID="82c7ec7c-f1db-4287-8ffa-9b3cd581f991" containerID="4eae82dd4dd49019ca1d1602a8d171685a5850d3955c9d167b971fdf3337348d" exitCode=0 Jul 07 18:21:09.216161 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:09.215724 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" Jul 07 18:21:09.216161 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:09.215747 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" event={"ID":"82c7ec7c-f1db-4287-8ffa-9b3cd581f991","Type":"ContainerDied","Data":"4eae82dd4dd49019ca1d1602a8d171685a5850d3955c9d167b971fdf3337348d"} Jul 07 18:21:09.216161 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:09.215799 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78" event={"ID":"82c7ec7c-f1db-4287-8ffa-9b3cd581f991","Type":"ContainerDied","Data":"1158e1388a978037a3c570250f48763e49ef96b0bb55f78663679287468b780d"} Jul 07 18:21:09.216161 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:09.215830 2583 scope.go:117] "RemoveContainer" containerID="4eae82dd4dd49019ca1d1602a8d171685a5850d3955c9d167b971fdf3337348d" Jul 07 18:21:09.225340 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:09.225322 2583 scope.go:117] "RemoveContainer" containerID="4eae82dd4dd49019ca1d1602a8d171685a5850d3955c9d167b971fdf3337348d" Jul 07 18:21:09.225644 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:21:09.225621 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eae82dd4dd49019ca1d1602a8d171685a5850d3955c9d167b971fdf3337348d\": container with ID starting with 4eae82dd4dd49019ca1d1602a8d171685a5850d3955c9d167b971fdf3337348d not found: ID does not exist" containerID="4eae82dd4dd49019ca1d1602a8d171685a5850d3955c9d167b971fdf3337348d" Jul 07 18:21:09.225740 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:09.225651 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eae82dd4dd49019ca1d1602a8d171685a5850d3955c9d167b971fdf3337348d"} err="failed to get container status \"4eae82dd4dd49019ca1d1602a8d171685a5850d3955c9d167b971fdf3337348d\": rpc error: code = NotFound desc = could not find container \"4eae82dd4dd49019ca1d1602a8d171685a5850d3955c9d167b971fdf3337348d\": container with ID starting with 4eae82dd4dd49019ca1d1602a8d171685a5850d3955c9d167b971fdf3337348d not found: ID does not exist" Jul 07 18:21:09.242383 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:09.242348 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78"] Jul 07 18:21:09.245123 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:09.245098 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d7f738e5-openshift-default-7556f54f7c-4lq78"] Jul 07 18:21:09.434428 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:09.434387 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82c7ec7c-f1db-4287-8ffa-9b3cd581f991" path="/var/lib/kubelet/pods/82c7ec7c-f1db-4287-8ffa-9b3cd581f991/volumes" Jul 07 18:21:09.916638 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:09.916593 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:21:10.064995 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.064959 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:21:10.137377 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.137346 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4"] Jul 07 18:21:10.137741 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.137728 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a" containerName="maas-api" Jul 07 18:21:10.137788 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.137743 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a" containerName="maas-api" Jul 07 18:21:10.137788 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.137756 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82c7ec7c-f1db-4287-8ffa-9b3cd581f991" containerName="istio-proxy" Jul 07 18:21:10.137788 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.137762 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="82c7ec7c-f1db-4287-8ffa-9b3cd581f991" containerName="istio-proxy" Jul 07 18:21:10.137883 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.137814 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4149f3b8-7bd2-4d44-9d0b-7ba7b7d2472a" containerName="maas-api" Jul 07 18:21:10.137883 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.137826 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="82c7ec7c-f1db-4287-8ffa-9b3cd581f991" containerName="istio-proxy" Jul 07 18:21:10.140638 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.140616 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:10.144336 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.144315 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-a814dbfb-openshift-default-dockercfg-glwxd\"" Jul 07 18:21:10.159080 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.159051 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4"] Jul 07 18:21:10.173203 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.173121 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-istio-podinfo\") pod \"e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:10.173203 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.173167 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-istio-data\") pod \"e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:10.173203 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.173189 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-workload-socket\") pod \"e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:10.173445 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.173295 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-credential-socket\") pod \"e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:10.173445 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.173322 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfxdw\" (UniqueName: \"kubernetes.io/projected/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-kube-api-access-nfxdw\") pod \"e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:10.173445 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.173355 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-workload-certs\") pod \"e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:10.173445 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.173402 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-istiod-ca-cert\") pod \"e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:10.173445 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.173429 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-istio-token\") pod \"e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:10.173618 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.173472 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-istio-envoy\") pod \"e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:10.182157 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.182114 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:21:10.274097 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.274060 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-istio-podinfo\") pod \"e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:10.274610 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.274110 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-istio-data\") pod \"e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:10.274610 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.274133 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-workload-socket\") pod \"e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:10.274610 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.274181 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-credential-socket\") pod \"e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:10.274610 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.274199 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfxdw\" (UniqueName: \"kubernetes.io/projected/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-kube-api-access-nfxdw\") pod \"e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:10.274610 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.274226 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-workload-certs\") pod \"e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:10.274610 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.274249 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-istiod-ca-cert\") pod \"e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:10.274610 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.274285 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-istio-token\") pod \"e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:10.274610 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.274317 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-istio-envoy\") pod \"e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:10.275011 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.274695 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-workload-certs\") pod \"e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:10.275011 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.274758 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-istio-data\") pod \"e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:10.275011 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.274957 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-workload-socket\") pod \"e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:10.275011 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.274964 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-credential-socket\") pod \"e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:10.275826 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.275772 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-istiod-ca-cert\") pod \"e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:10.277352 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.277323 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-istio-podinfo\") pod \"e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:10.277664 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.277643 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-istio-envoy\") pod \"e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:10.277745 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.277682 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:21:10.286457 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.286423 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfxdw\" (UniqueName: \"kubernetes.io/projected/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-kube-api-access-nfxdw\") pod \"e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:10.286457 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.286449 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-istio-token\") pod \"e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:10.450834 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.450744 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:10.599760 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.599721 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4"] Jul 07 18:21:10.607483 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.607448 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:21:10.607576 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.607524 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:21:10.607615 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:10.607579 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:21:11.229761 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:11.229724 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" event={"ID":"d909785b-3c0f-47f2-9bf7-877b7c4c8b13","Type":"ContainerStarted","Data":"9f2fb72a0cc913c2450a22511a55309495dd56d4edf064e5e4b95ce9a22310ea"} Jul 07 18:21:11.229988 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:11.229778 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" event={"ID":"d909785b-3c0f-47f2-9bf7-877b7c4c8b13","Type":"ContainerStarted","Data":"3da1e722e9031908bd78fc4c5243619766e9c4ca87240945a467490634c0f6dc"} Jul 07 18:21:11.252443 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:11.252384 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" podStartSLOduration=1.252365858 podStartE2EDuration="1.252365858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:21:11.251112048 +0000 UTC m=+1620.333715752" watchObservedRunningTime="2026-07-07 18:21:11.252365858 +0000 UTC m=+1620.334969647" Jul 07 18:21:11.378952 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:11.378879 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:21:11.451667 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:11.451627 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:11.457018 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:11.456989 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:11.479141 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:11.479100 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:21:11.504730 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:11.504698 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7"] Jul 07 18:21:11.508943 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:11.508894 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7" Jul 07 18:21:11.511651 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:11.511627 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-a814dbfb-serving-cert\"" Jul 07 18:21:11.511796 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:11.511670 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-jnv2d\"" Jul 07 18:21:11.520040 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:11.520009 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7"] Jul 07 18:21:11.587734 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:11.587692 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sck7\" (UniqueName: \"kubernetes.io/projected/03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa-kube-api-access-8sck7\") pod \"maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7\" (UID: \"03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa\") " pod="opendatahub/maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7" Jul 07 18:21:11.587956 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:11.587743 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa-maas-api-tls\") pod \"maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7\" (UID: \"03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa\") " pod="opendatahub/maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7" Jul 07 18:21:11.587956 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:11.587803 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa-openshift-service-ca\") pod \"maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7\" (UID: \"03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa\") " pod="opendatahub/maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7" Jul 07 18:21:11.593031 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:11.593000 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:21:11.688730 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:11.688686 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sck7\" (UniqueName: \"kubernetes.io/projected/03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa-kube-api-access-8sck7\") pod \"maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7\" (UID: \"03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa\") " pod="opendatahub/maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7" Jul 07 18:21:11.688931 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:11.688753 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa-maas-api-tls\") pod \"maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7\" (UID: \"03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa\") " pod="opendatahub/maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7" Jul 07 18:21:11.688931 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:11.688808 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa-openshift-service-ca\") pod \"maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7\" (UID: \"03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa\") " pod="opendatahub/maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7" Jul 07 18:21:11.689064 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:21:11.688941 2583 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-a814dbfb-serving-cert: secret "maas-api-e2e-derive-a814dbfb-serving-cert" not found Jul 07 18:21:11.689064 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:21:11.689013 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa-maas-api-tls podName:03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa nodeName:}" failed. No retries permitted until 2026-07-07 18:21:12.188989175 +0000 UTC m=+1621.271592860 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa-maas-api-tls") pod "maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7" (UID: "03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa") : secret "maas-api-e2e-derive-a814dbfb-serving-cert" not found Jul 07 18:21:11.689622 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:11.689596 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa-openshift-service-ca\") pod \"maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7\" (UID: \"03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa\") " pod="opendatahub/maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7" Jul 07 18:21:11.697881 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:11.697854 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sck7\" (UniqueName: \"kubernetes.io/projected/03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa-kube-api-access-8sck7\") pod \"maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7\" (UID: \"03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa\") " pod="opendatahub/maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7" Jul 07 18:21:11.785881 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:11.785778 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:21:12.153243 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:12.153208 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:21:12.194112 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:12.194077 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa-maas-api-tls\") pod \"maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7\" (UID: \"03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa\") " pod="opendatahub/maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7" Jul 07 18:21:12.196777 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:12.196752 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa-maas-api-tls\") pod \"maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7\" (UID: \"03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa\") " pod="opendatahub/maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7" Jul 07 18:21:12.234782 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:12.234755 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:12.235865 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:12.235842 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:12.253798 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:12.253720 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:21:12.423793 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:12.423699 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7" Jul 07 18:21:12.568808 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:12.568782 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7"] Jul 07 18:21:12.571109 ip-10-0-136-192 kubenswrapper[2583]: W0707 18:21:12.571077 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b76ba5_9b8e_4c02_bf5c_b45d3659f2fa.slice/crio-7b9e93612652c13b88b6dec0f2657daeca22d4985a2b97f0f493d729a0636173 WatchSource:0}: Error finding container 7b9e93612652c13b88b6dec0f2657daeca22d4985a2b97f0f493d729a0636173: Status 404 returned error can't find the container with id 7b9e93612652c13b88b6dec0f2657daeca22d4985a2b97f0f493d729a0636173 Jul 07 18:21:13.240700 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:13.240669 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7" event={"ID":"03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa","Type":"ContainerStarted","Data":"363bd10d3344dbe552b4ae770b26817d97bdda790d6609ea902db00652bfdb46"} Jul 07 18:21:13.240888 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:13.240860 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7" Jul 07 18:21:13.240888 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:13.240880 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7" event={"ID":"03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa","Type":"ContainerStarted","Data":"7b9e93612652c13b88b6dec0f2657daeca22d4985a2b97f0f493d729a0636173"} Jul 07 18:21:13.261953 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:13.261886 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7" podStartSLOduration=1.740441846 podStartE2EDuration="2.261867659s" totalImagesPullingTime="521.425813ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:21:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:21:13.261061594 +0000 UTC m=+1622.343665299" watchObservedRunningTime="2026-07-07 18:21:13.261867659 +0000 UTC m=+1622.344471363" Jul 07 18:21:13.987146 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:13.987099 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:21:15.737411 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:15.737368 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:21:15.934354 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:15.934317 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4"] Jul 07 18:21:15.934555 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:15.934494 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" podUID="d909785b-3c0f-47f2-9bf7-877b7c4c8b13" containerName="istio-proxy" containerID="cri-o://9f2fb72a0cc913c2450a22511a55309495dd56d4edf064e5e4b95ce9a22310ea" gracePeriod=30 Jul 07 18:21:15.950485 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:15.950450 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:21:16.065252 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:16.065221 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:21:16.164484 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:16.164451 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7"] Jul 07 18:21:16.164652 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:16.164626 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7" podUID="03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa" containerName="maas-api" containerID="cri-o://363bd10d3344dbe552b4ae770b26817d97bdda790d6609ea902db00652bfdb46" gracePeriod=30 Jul 07 18:21:16.186373 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:21:16.186343 2583 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b76ba5_9b8e_4c02_bf5c_b45d3659f2fa.slice/crio-363bd10d3344dbe552b4ae770b26817d97bdda790d6609ea902db00652bfdb46.scope\": RecentStats: unable to find data in memory cache]" Jul 07 18:21:16.205296 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:16.205263 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:21:16.427711 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:16.427687 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7" Jul 07 18:21:16.548108 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:16.548072 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa-openshift-service-ca\") pod \"03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa\" (UID: \"03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa\") " Jul 07 18:21:16.548287 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:16.548124 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa-maas-api-tls\") pod \"03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa\" (UID: \"03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa\") " Jul 07 18:21:16.548287 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:16.548165 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sck7\" (UniqueName: \"kubernetes.io/projected/03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa-kube-api-access-8sck7\") pod \"03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa\" (UID: \"03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa\") " Jul 07 18:21:16.548513 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:16.548480 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa" (UID: "03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:21:16.548657 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:16.548637 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa-openshift-service-ca\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:21:16.550552 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:16.550527 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa-kube-api-access-8sck7" (OuterVolumeSpecName: "kube-api-access-8sck7") pod "03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa" (UID: "03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa"). InnerVolumeSpecName "kube-api-access-8sck7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:21:16.550686 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:16.550513 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa" (UID: "03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:21:16.649727 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:16.649635 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa-maas-api-tls\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:21:16.649727 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:16.649667 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sck7\" (UniqueName: \"kubernetes.io/projected/03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa-kube-api-access-8sck7\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:21:17.261649 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:17.261609 2583 generic.go:356] "Generic (PLEG): container finished" podID="03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa" containerID="363bd10d3344dbe552b4ae770b26817d97bdda790d6609ea902db00652bfdb46" exitCode=0 Jul 07 18:21:17.262090 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:17.261662 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7" Jul 07 18:21:17.262090 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:17.261696 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7" event={"ID":"03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa","Type":"ContainerDied","Data":"363bd10d3344dbe552b4ae770b26817d97bdda790d6609ea902db00652bfdb46"} Jul 07 18:21:17.262090 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:17.261756 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7" event={"ID":"03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa","Type":"ContainerDied","Data":"7b9e93612652c13b88b6dec0f2657daeca22d4985a2b97f0f493d729a0636173"} Jul 07 18:21:17.262090 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:17.261778 2583 scope.go:117] "RemoveContainer" containerID="363bd10d3344dbe552b4ae770b26817d97bdda790d6609ea902db00652bfdb46" Jul 07 18:21:17.271502 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:17.271483 2583 scope.go:117] "RemoveContainer" containerID="363bd10d3344dbe552b4ae770b26817d97bdda790d6609ea902db00652bfdb46" Jul 07 18:21:17.271795 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:21:17.271773 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"363bd10d3344dbe552b4ae770b26817d97bdda790d6609ea902db00652bfdb46\": container with ID starting with 363bd10d3344dbe552b4ae770b26817d97bdda790d6609ea902db00652bfdb46 not found: ID does not exist" containerID="363bd10d3344dbe552b4ae770b26817d97bdda790d6609ea902db00652bfdb46" Jul 07 18:21:17.271872 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:17.271802 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"363bd10d3344dbe552b4ae770b26817d97bdda790d6609ea902db00652bfdb46"} err="failed to get container status \"363bd10d3344dbe552b4ae770b26817d97bdda790d6609ea902db00652bfdb46\": rpc error: code = NotFound desc = could not find container \"363bd10d3344dbe552b4ae770b26817d97bdda790d6609ea902db00652bfdb46\": container with ID starting with 363bd10d3344dbe552b4ae770b26817d97bdda790d6609ea902db00652bfdb46 not found: ID does not exist" Jul 07 18:21:17.326271 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:17.326236 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7"] Jul 07 18:21:17.353270 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:17.353229 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-a814dbfb-b4975c5b7-mddc7"] Jul 07 18:21:17.429318 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:17.429282 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa" path="/var/lib/kubelet/pods/03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa/volumes" Jul 07 18:21:18.238498 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:18.238456 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:21:21.191881 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.191855 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:21.285586 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.285488 2583 generic.go:356] "Generic (PLEG): container finished" podID="d909785b-3c0f-47f2-9bf7-877b7c4c8b13" containerID="9f2fb72a0cc913c2450a22511a55309495dd56d4edf064e5e4b95ce9a22310ea" exitCode=0 Jul 07 18:21:21.285586 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.285536 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" event={"ID":"d909785b-3c0f-47f2-9bf7-877b7c4c8b13","Type":"ContainerDied","Data":"9f2fb72a0cc913c2450a22511a55309495dd56d4edf064e5e4b95ce9a22310ea"} Jul 07 18:21:21.285774 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.285610 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" event={"ID":"d909785b-3c0f-47f2-9bf7-877b7c4c8b13","Type":"ContainerDied","Data":"3da1e722e9031908bd78fc4c5243619766e9c4ca87240945a467490634c0f6dc"} Jul 07 18:21:21.285774 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.285535 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4" Jul 07 18:21:21.285774 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.285632 2583 scope.go:117] "RemoveContainer" containerID="9f2fb72a0cc913c2450a22511a55309495dd56d4edf064e5e4b95ce9a22310ea" Jul 07 18:21:21.294290 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.294270 2583 scope.go:117] "RemoveContainer" containerID="9f2fb72a0cc913c2450a22511a55309495dd56d4edf064e5e4b95ce9a22310ea" Jul 07 18:21:21.294559 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:21:21.294534 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f2fb72a0cc913c2450a22511a55309495dd56d4edf064e5e4b95ce9a22310ea\": container with ID starting with 9f2fb72a0cc913c2450a22511a55309495dd56d4edf064e5e4b95ce9a22310ea not found: ID does not exist" containerID="9f2fb72a0cc913c2450a22511a55309495dd56d4edf064e5e4b95ce9a22310ea" Jul 07 18:21:21.294611 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.294574 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f2fb72a0cc913c2450a22511a55309495dd56d4edf064e5e4b95ce9a22310ea"} err="failed to get container status \"9f2fb72a0cc913c2450a22511a55309495dd56d4edf064e5e4b95ce9a22310ea\": rpc error: code = NotFound desc = could not find container \"9f2fb72a0cc913c2450a22511a55309495dd56d4edf064e5e4b95ce9a22310ea\": container with ID starting with 9f2fb72a0cc913c2450a22511a55309495dd56d4edf064e5e4b95ce9a22310ea not found: ID does not exist" Jul 07 18:21:21.295871 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.295852 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-credential-socket\") pod \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " Jul 07 18:21:21.296001 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.295949 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfxdw\" (UniqueName: \"kubernetes.io/projected/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-kube-api-access-nfxdw\") pod \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " Jul 07 18:21:21.296001 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.295989 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-istiod-ca-cert\") pod \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " Jul 07 18:21:21.296109 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.296021 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-workload-certs\") pod \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " Jul 07 18:21:21.296109 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.296049 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-workload-socket\") pod \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " Jul 07 18:21:21.296109 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.296086 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-istio-data\") pod \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " Jul 07 18:21:21.296251 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.296126 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d909785b-3c0f-47f2-9bf7-877b7c4c8b13" (UID: "d909785b-3c0f-47f2-9bf7-877b7c4c8b13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:21:21.296251 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.296133 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-istio-token\") pod \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " Jul 07 18:21:21.296251 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.296202 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-istio-podinfo\") pod \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " Jul 07 18:21:21.296408 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.296293 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-istio-envoy\") pod \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\" (UID: \"d909785b-3c0f-47f2-9bf7-877b7c4c8b13\") " Jul 07 18:21:21.296408 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.296328 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d909785b-3c0f-47f2-9bf7-877b7c4c8b13" (UID: "d909785b-3c0f-47f2-9bf7-877b7c4c8b13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:21:21.296503 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.296411 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d909785b-3c0f-47f2-9bf7-877b7c4c8b13" (UID: "d909785b-3c0f-47f2-9bf7-877b7c4c8b13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:21:21.296503 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.296445 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d909785b-3c0f-47f2-9bf7-877b7c4c8b13" (UID: "d909785b-3c0f-47f2-9bf7-877b7c4c8b13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:21:21.296609 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.296512 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-istio-data" (OuterVolumeSpecName: "istio-data") pod "d909785b-3c0f-47f2-9bf7-877b7c4c8b13" (UID: "d909785b-3c0f-47f2-9bf7-877b7c4c8b13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:21:21.296692 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.296673 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-istio-data\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:21:21.296747 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.296695 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-credential-socket\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:21:21.296747 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.296705 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-istiod-ca-cert\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:21:21.296747 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.296714 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-workload-certs\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:21:21.296747 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.296722 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-workload-socket\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:21:21.298394 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.298371 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-istio-token" (OuterVolumeSpecName: "istio-token") pod "d909785b-3c0f-47f2-9bf7-877b7c4c8b13" (UID: "d909785b-3c0f-47f2-9bf7-877b7c4c8b13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:21:21.298506 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.298472 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d909785b-3c0f-47f2-9bf7-877b7c4c8b13" (UID: "d909785b-3c0f-47f2-9bf7-877b7c4c8b13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 18:21:21.298836 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.298809 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-kube-api-access-nfxdw" (OuterVolumeSpecName: "kube-api-access-nfxdw") pod "d909785b-3c0f-47f2-9bf7-877b7c4c8b13" (UID: "d909785b-3c0f-47f2-9bf7-877b7c4c8b13"). InnerVolumeSpecName "kube-api-access-nfxdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:21:21.298977 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.298843 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d909785b-3c0f-47f2-9bf7-877b7c4c8b13" (UID: "d909785b-3c0f-47f2-9bf7-877b7c4c8b13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:21:21.398077 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.398039 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfxdw\" (UniqueName: \"kubernetes.io/projected/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-kube-api-access-nfxdw\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:21:21.398077 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.398073 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-istio-token\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:21:21.398077 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.398085 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-istio-podinfo\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:21:21.398329 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.398093 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d909785b-3c0f-47f2-9bf7-877b7c4c8b13-istio-envoy\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:21:21.604896 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.604859 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4"] Jul 07 18:21:21.608500 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:21.608468 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-a814dbfb-openshift-default-6dbf5f48c9-nc4f4"] Jul 07 18:21:22.227266 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:22.227232 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:21:22.307292 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:22.307252 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:21:22.816647 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:22.816610 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:21:22.914737 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:22.914706 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:21:23.429307 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:23.429275 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d909785b-3c0f-47f2-9bf7-877b7c4c8b13" path="/var/lib/kubelet/pods/d909785b-3c0f-47f2-9bf7-877b7c4c8b13/volumes" Jul 07 18:21:24.408739 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:24.408709 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:21:25.771982 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:25.771939 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:21:30.798236 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:30.798182 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:21:30.918546 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:30.918506 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:21:31.013098 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:31.013061 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:21:31.136739 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:31.136659 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:21:31.588336 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:31.588303 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:21:35.962280 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:35.962245 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:21:57.597666 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:21:57.597633 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:22:04.975369 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:22:04.975336 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:22:06.544553 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:22:06.544522 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:22:21.260927 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:22:21.260863 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:22:28.442593 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:22:28.442553 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:22:28.512164 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:22:28.512133 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:22:28.609655 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:22:28.609617 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:22:28.706516 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:22:28.706424 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:22:39.318711 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:22:39.318624 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:22:40.106572 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:22:40.106533 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:22:46.594243 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:22:46.594198 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:22:46.742968 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:22:46.742929 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:22:46.878296 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:22:46.878212 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:22:47.371607 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:22:47.371571 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:22:47.824171 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:22:47.824135 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:22:47.918748 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:22:47.918712 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:22:48.019686 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:22:48.019649 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:22:49.887954 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:22:49.887921 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:22:50.063883 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:22:50.063844 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:22:56.569368 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:22:56.569330 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:22:56.666889 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:22:56.666834 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:22:57.296782 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:22:57.296748 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:22:59.639785 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:22:59.639747 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:23:04.019501 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.019464 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:04.117894 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.117849 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:04.304630 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.304598 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:04.754144 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.754054 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:04.768412 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.768381 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs"] Jul 07 18:23:04.768814 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.768800 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d909785b-3c0f-47f2-9bf7-877b7c4c8b13" containerName="istio-proxy" Jul 07 18:23:04.768881 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.768817 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d909785b-3c0f-47f2-9bf7-877b7c4c8b13" containerName="istio-proxy" Jul 07 18:23:04.768881 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.768828 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa" containerName="maas-api" Jul 07 18:23:04.768881 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.768834 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa" containerName="maas-api" Jul 07 18:23:04.769015 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.768893 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="03b76ba5-9b8e-4c02-bf5c-b45d3659f2fa" containerName="maas-api" Jul 07 18:23:04.769015 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.768924 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d909785b-3c0f-47f2-9bf7-877b7c4c8b13" containerName="istio-proxy" Jul 07 18:23:04.771046 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.771029 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:04.778596 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.778574 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f8b2cd9b-openshift-default-dockercfg-ndthg\"" Jul 07 18:23:04.821506 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.821464 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs"] Jul 07 18:23:04.893073 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.893028 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a104662b-e5e4-43c6-b4dd-455c0100a176-workload-socket\") pod \"e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:04.893073 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.893077 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a104662b-e5e4-43c6-b4dd-455c0100a176-istio-envoy\") pod \"e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:04.893310 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.893094 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a104662b-e5e4-43c6-b4dd-455c0100a176-istio-podinfo\") pod \"e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:04.893310 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.893121 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a104662b-e5e4-43c6-b4dd-455c0100a176-istio-token\") pod \"e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:04.893310 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.893141 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kj5f\" (UniqueName: \"kubernetes.io/projected/a104662b-e5e4-43c6-b4dd-455c0100a176-kube-api-access-4kj5f\") pod \"e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:04.893310 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.893197 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a104662b-e5e4-43c6-b4dd-455c0100a176-istiod-ca-cert\") pod \"e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:04.893310 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.893218 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a104662b-e5e4-43c6-b4dd-455c0100a176-istio-data\") pod \"e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:04.893310 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.893246 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a104662b-e5e4-43c6-b4dd-455c0100a176-workload-certs\") pod \"e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:04.893531 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.893353 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a104662b-e5e4-43c6-b4dd-455c0100a176-credential-socket\") pod \"e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:04.994533 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.994493 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a104662b-e5e4-43c6-b4dd-455c0100a176-credential-socket\") pod \"e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:04.994723 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.994567 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a104662b-e5e4-43c6-b4dd-455c0100a176-workload-socket\") pod \"e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:04.994723 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.994599 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a104662b-e5e4-43c6-b4dd-455c0100a176-istio-envoy\") pod \"e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:04.994723 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.994621 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a104662b-e5e4-43c6-b4dd-455c0100a176-istio-podinfo\") pod \"e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:04.994723 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.994646 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a104662b-e5e4-43c6-b4dd-455c0100a176-istio-token\") pod \"e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:04.994723 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.994672 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kj5f\" (UniqueName: \"kubernetes.io/projected/a104662b-e5e4-43c6-b4dd-455c0100a176-kube-api-access-4kj5f\") pod \"e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:04.994723 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.994723 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a104662b-e5e4-43c6-b4dd-455c0100a176-istiod-ca-cert\") pod \"e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:04.995109 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.994757 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a104662b-e5e4-43c6-b4dd-455c0100a176-istio-data\") pod \"e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:04.995109 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.994815 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a104662b-e5e4-43c6-b4dd-455c0100a176-workload-certs\") pod \"e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:04.995109 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.995007 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a104662b-e5e4-43c6-b4dd-455c0100a176-credential-socket\") pod \"e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:04.995256 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.995108 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a104662b-e5e4-43c6-b4dd-455c0100a176-workload-socket\") pod \"e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:04.995365 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.995347 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a104662b-e5e4-43c6-b4dd-455c0100a176-workload-certs\") pod \"e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:04.995419 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.995369 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a104662b-e5e4-43c6-b4dd-455c0100a176-istio-data\") pod \"e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:04.996041 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.996017 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a104662b-e5e4-43c6-b4dd-455c0100a176-istiod-ca-cert\") pod \"e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:04.997100 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.997079 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a104662b-e5e4-43c6-b4dd-455c0100a176-istio-envoy\") pod \"e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:04.997397 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:04.997378 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a104662b-e5e4-43c6-b4dd-455c0100a176-istio-podinfo\") pod \"e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:05.011288 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:05.011220 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a104662b-e5e4-43c6-b4dd-455c0100a176-istio-token\") pod \"e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:05.011419 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:05.011390 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kj5f\" (UniqueName: \"kubernetes.io/projected/a104662b-e5e4-43c6-b4dd-455c0100a176-kube-api-access-4kj5f\") pod \"e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:05.053467 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:05.053436 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:05.081520 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:05.081480 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:05.234453 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:05.234424 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs"] Jul 07 18:23:05.236425 ip-10-0-136-192 kubenswrapper[2583]: W0707 18:23:05.236387 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda104662b_e5e4_43c6_b4dd_455c0100a176.slice/crio-ef4ef54aff73063a36a8996c0b0096553a6e58297bd0ed2f7ecb9e2911b7def3 WatchSource:0}: Error finding container ef4ef54aff73063a36a8996c0b0096553a6e58297bd0ed2f7ecb9e2911b7def3: Status 404 returned error can't find the container with id ef4ef54aff73063a36a8996c0b0096553a6e58297bd0ed2f7ecb9e2911b7def3 Jul 07 18:23:05.238575 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:05.238538 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:23:05.238643 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:05.238621 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:23:05.238686 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:05.238657 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:23:05.800215 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:05.800182 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" event={"ID":"a104662b-e5e4-43c6-b4dd-455c0100a176","Type":"ContainerStarted","Data":"6f718f3512d4a01c9f8bbccd97c412184734011c2469d08bef621800297aeb02"} Jul 07 18:23:05.800404 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:05.800234 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" event={"ID":"a104662b-e5e4-43c6-b4dd-455c0100a176","Type":"ContainerStarted","Data":"ef4ef54aff73063a36a8996c0b0096553a6e58297bd0ed2f7ecb9e2911b7def3"} Jul 07 18:23:05.825389 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:05.825323 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" podStartSLOduration=1.82530473 podStartE2EDuration="1.82530473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:05.824579154 +0000 UTC m=+1734.907182860" watchObservedRunningTime="2026-07-07 18:23:05.82530473 +0000 UTC m=+1734.907908434" Jul 07 18:23:06.082194 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:06.082107 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:06.087695 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:06.087665 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:06.805210 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:06.805175 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:06.806180 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:06.806157 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:07.846549 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:07.846499 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:07.923834 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:07.923782 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:07.936447 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:07.936412 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb"] Jul 07 18:23:07.938964 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:07.938945 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb" Jul 07 18:23:07.941973 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:07.941949 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-f8b2cd9b-serving-cert\"" Jul 07 18:23:07.942126 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:07.941951 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-jnv2d\"" Jul 07 18:23:07.955317 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:07.955286 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb"] Jul 07 18:23:08.005819 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:08.005782 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:08.022024 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:08.021985 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/315ad7a9-9938-47e5-9ede-b3478b8a9ef2-openshift-service-ca\") pod \"maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb\" (UID: \"315ad7a9-9938-47e5-9ede-b3478b8a9ef2\") " pod="opendatahub/maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb" Jul 07 18:23:08.022230 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:08.022117 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/315ad7a9-9938-47e5-9ede-b3478b8a9ef2-maas-api-tls\") pod \"maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb\" (UID: \"315ad7a9-9938-47e5-9ede-b3478b8a9ef2\") " pod="opendatahub/maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb" Jul 07 18:23:08.022230 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:08.022225 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm5zz\" (UniqueName: \"kubernetes.io/projected/315ad7a9-9938-47e5-9ede-b3478b8a9ef2-kube-api-access-lm5zz\") pod \"maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb\" (UID: \"315ad7a9-9938-47e5-9ede-b3478b8a9ef2\") " pod="opendatahub/maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb" Jul 07 18:23:08.108468 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:08.108381 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:08.122738 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:08.122699 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm5zz\" (UniqueName: \"kubernetes.io/projected/315ad7a9-9938-47e5-9ede-b3478b8a9ef2-kube-api-access-lm5zz\") pod \"maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb\" (UID: \"315ad7a9-9938-47e5-9ede-b3478b8a9ef2\") " pod="opendatahub/maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb" Jul 07 18:23:08.122945 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:08.122764 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/315ad7a9-9938-47e5-9ede-b3478b8a9ef2-openshift-service-ca\") pod \"maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb\" (UID: \"315ad7a9-9938-47e5-9ede-b3478b8a9ef2\") " pod="opendatahub/maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb" Jul 07 18:23:08.122945 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:08.122845 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/315ad7a9-9938-47e5-9ede-b3478b8a9ef2-maas-api-tls\") pod \"maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb\" (UID: \"315ad7a9-9938-47e5-9ede-b3478b8a9ef2\") " pod="opendatahub/maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb" Jul 07 18:23:08.123382 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:08.123361 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/315ad7a9-9938-47e5-9ede-b3478b8a9ef2-openshift-service-ca\") pod \"maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb\" (UID: \"315ad7a9-9938-47e5-9ede-b3478b8a9ef2\") " pod="opendatahub/maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb" Jul 07 18:23:08.125698 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:08.125672 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/315ad7a9-9938-47e5-9ede-b3478b8a9ef2-maas-api-tls\") pod \"maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb\" (UID: \"315ad7a9-9938-47e5-9ede-b3478b8a9ef2\") " pod="opendatahub/maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb" Jul 07 18:23:08.137460 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:08.137430 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm5zz\" (UniqueName: \"kubernetes.io/projected/315ad7a9-9938-47e5-9ede-b3478b8a9ef2-kube-api-access-lm5zz\") pod \"maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb\" (UID: \"315ad7a9-9938-47e5-9ede-b3478b8a9ef2\") " pod="opendatahub/maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb" Jul 07 18:23:08.249928 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:08.249884 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb" Jul 07 18:23:08.403554 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:08.403527 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb"] Jul 07 18:23:08.405999 ip-10-0-136-192 kubenswrapper[2583]: W0707 18:23:08.405970 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod315ad7a9_9938_47e5_9ede_b3478b8a9ef2.slice/crio-10f8aa2ca493f65b3958715bb8db9d8dc981bdefa562175155388e4454718b1d WatchSource:0}: Error finding container 10f8aa2ca493f65b3958715bb8db9d8dc981bdefa562175155388e4454718b1d: Status 404 returned error can't find the container with id 10f8aa2ca493f65b3958715bb8db9d8dc981bdefa562175155388e4454718b1d Jul 07 18:23:08.816125 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:08.816089 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb" event={"ID":"315ad7a9-9938-47e5-9ede-b3478b8a9ef2","Type":"ContainerStarted","Data":"10f8aa2ca493f65b3958715bb8db9d8dc981bdefa562175155388e4454718b1d"} Jul 07 18:23:08.935541 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:08.935514 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:09.083937 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:09.083828 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:09.821910 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:09.821873 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb" event={"ID":"315ad7a9-9938-47e5-9ede-b3478b8a9ef2","Type":"ContainerStarted","Data":"637c1ac713b39fd6b7db7a4264cc0c3cc9642c852c741ef8d50b6a1506a9e4f6"} Jul 07 18:23:09.822096 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:09.822031 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb" Jul 07 18:23:09.843639 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:09.843587 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb" podStartSLOduration=2.331448198 podStartE2EDuration="2.843568735s" totalImagesPullingTime="512.120537ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:23:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:09.841511513 +0000 UTC m=+1738.924115219" watchObservedRunningTime="2026-07-07 18:23:09.843568735 +0000 UTC m=+1738.926172439" Jul 07 18:23:12.413413 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:12.413376 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:13.169492 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:13.169445 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:13.265038 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:13.265006 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:13.658452 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:13.658418 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:13.892215 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:13.892179 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:23:15.832376 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:15.832346 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb" Jul 07 18:23:19.937821 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:19.937774 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:20.011026 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:20.010990 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:20.366477 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:20.366446 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:20.793080 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:20.793049 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:20.836114 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:20.836080 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb"] Jul 07 18:23:20.836407 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:20.836376 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb" podUID="315ad7a9-9938-47e5-9ede-b3478b8a9ef2" containerName="maas-api" containerID="cri-o://637c1ac713b39fd6b7db7a4264cc0c3cc9642c852c741ef8d50b6a1506a9e4f6" gracePeriod=30 Jul 07 18:23:20.864580 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:20.864549 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:21.092382 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:21.092355 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb" Jul 07 18:23:21.260129 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:21.260085 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/315ad7a9-9938-47e5-9ede-b3478b8a9ef2-openshift-service-ca\") pod \"315ad7a9-9938-47e5-9ede-b3478b8a9ef2\" (UID: \"315ad7a9-9938-47e5-9ede-b3478b8a9ef2\") " Jul 07 18:23:21.260286 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:21.260146 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/315ad7a9-9938-47e5-9ede-b3478b8a9ef2-maas-api-tls\") pod \"315ad7a9-9938-47e5-9ede-b3478b8a9ef2\" (UID: \"315ad7a9-9938-47e5-9ede-b3478b8a9ef2\") " Jul 07 18:23:21.260286 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:21.260204 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm5zz\" (UniqueName: \"kubernetes.io/projected/315ad7a9-9938-47e5-9ede-b3478b8a9ef2-kube-api-access-lm5zz\") pod \"315ad7a9-9938-47e5-9ede-b3478b8a9ef2\" (UID: \"315ad7a9-9938-47e5-9ede-b3478b8a9ef2\") " Jul 07 18:23:21.260537 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:21.260501 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/315ad7a9-9938-47e5-9ede-b3478b8a9ef2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "315ad7a9-9938-47e5-9ede-b3478b8a9ef2" (UID: "315ad7a9-9938-47e5-9ede-b3478b8a9ef2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:23:21.262390 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:21.262367 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/315ad7a9-9938-47e5-9ede-b3478b8a9ef2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "315ad7a9-9938-47e5-9ede-b3478b8a9ef2" (UID: "315ad7a9-9938-47e5-9ede-b3478b8a9ef2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:23:21.262523 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:21.262505 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/315ad7a9-9938-47e5-9ede-b3478b8a9ef2-kube-api-access-lm5zz" (OuterVolumeSpecName: "kube-api-access-lm5zz") pod "315ad7a9-9938-47e5-9ede-b3478b8a9ef2" (UID: "315ad7a9-9938-47e5-9ede-b3478b8a9ef2"). InnerVolumeSpecName "kube-api-access-lm5zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:23:21.361293 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:21.361198 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/315ad7a9-9938-47e5-9ede-b3478b8a9ef2-openshift-service-ca\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:23:21.361293 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:21.361234 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/315ad7a9-9938-47e5-9ede-b3478b8a9ef2-maas-api-tls\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:23:21.361293 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:21.361247 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm5zz\" (UniqueName: \"kubernetes.io/projected/315ad7a9-9938-47e5-9ede-b3478b8a9ef2-kube-api-access-lm5zz\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:23:21.880023 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:21.879984 2583 generic.go:356] "Generic (PLEG): container finished" podID="315ad7a9-9938-47e5-9ede-b3478b8a9ef2" containerID="637c1ac713b39fd6b7db7a4264cc0c3cc9642c852c741ef8d50b6a1506a9e4f6" exitCode=0 Jul 07 18:23:21.880182 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:21.880050 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb" Jul 07 18:23:21.880182 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:21.880070 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb" event={"ID":"315ad7a9-9938-47e5-9ede-b3478b8a9ef2","Type":"ContainerDied","Data":"637c1ac713b39fd6b7db7a4264cc0c3cc9642c852c741ef8d50b6a1506a9e4f6"} Jul 07 18:23:21.880182 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:21.880125 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb" event={"ID":"315ad7a9-9938-47e5-9ede-b3478b8a9ef2","Type":"ContainerDied","Data":"10f8aa2ca493f65b3958715bb8db9d8dc981bdefa562175155388e4454718b1d"} Jul 07 18:23:21.880182 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:21.880150 2583 scope.go:117] "RemoveContainer" containerID="637c1ac713b39fd6b7db7a4264cc0c3cc9642c852c741ef8d50b6a1506a9e4f6" Jul 07 18:23:21.889404 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:21.889377 2583 scope.go:117] "RemoveContainer" containerID="637c1ac713b39fd6b7db7a4264cc0c3cc9642c852c741ef8d50b6a1506a9e4f6" Jul 07 18:23:21.889768 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:23:21.889741 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"637c1ac713b39fd6b7db7a4264cc0c3cc9642c852c741ef8d50b6a1506a9e4f6\": container with ID starting with 637c1ac713b39fd6b7db7a4264cc0c3cc9642c852c741ef8d50b6a1506a9e4f6 not found: ID does not exist" containerID="637c1ac713b39fd6b7db7a4264cc0c3cc9642c852c741ef8d50b6a1506a9e4f6" Jul 07 18:23:21.889835 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:21.889783 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"637c1ac713b39fd6b7db7a4264cc0c3cc9642c852c741ef8d50b6a1506a9e4f6"} err="failed to get container status \"637c1ac713b39fd6b7db7a4264cc0c3cc9642c852c741ef8d50b6a1506a9e4f6\": rpc error: code = NotFound desc = could not find container \"637c1ac713b39fd6b7db7a4264cc0c3cc9642c852c741ef8d50b6a1506a9e4f6\": container with ID starting with 637c1ac713b39fd6b7db7a4264cc0c3cc9642c852c741ef8d50b6a1506a9e4f6 not found: ID does not exist" Jul 07 18:23:21.933347 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:21.933312 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb"] Jul 07 18:23:21.944041 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:21.944008 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-f8b2cd9b-5cdc67b769-lrccb"] Jul 07 18:23:22.167723 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:22.167693 2583 patch_prober.go:28] interesting pod/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": read tcp 10.133.0.2:36740->10.133.0.35:15021: read: connection reset by peer" start-of-body= Jul 07 18:23:22.168153 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:22.167761 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" podUID="a104662b-e5e4-43c6-b4dd-455c0100a176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": read tcp 10.133.0.2:36740->10.133.0.35:15021: read: connection reset by peer" Jul 07 18:23:22.887127 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:22.887089 2583 generic.go:356] "Generic (PLEG): container finished" podID="a104662b-e5e4-43c6-b4dd-455c0100a176" containerID="6f718f3512d4a01c9f8bbccd97c412184734011c2469d08bef621800297aeb02" exitCode=0 Jul 07 18:23:22.887340 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:22.887191 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" event={"ID":"a104662b-e5e4-43c6-b4dd-455c0100a176","Type":"ContainerDied","Data":"6f718f3512d4a01c9f8bbccd97c412184734011c2469d08bef621800297aeb02"} Jul 07 18:23:22.887743 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:22.887726 2583 scope.go:117] "RemoveContainer" containerID="6f718f3512d4a01c9f8bbccd97c412184734011c2469d08bef621800297aeb02" Jul 07 18:23:22.888997 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:22.888969 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:23:22.889121 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:22.889041 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:23:22.889121 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:22.889073 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:23:23.430485 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:23.430443 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="315ad7a9-9938-47e5-9ede-b3478b8a9ef2" path="/var/lib/kubelet/pods/315ad7a9-9938-47e5-9ede-b3478b8a9ef2/volumes" Jul 07 18:23:23.899634 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:23.899587 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" event={"ID":"a104662b-e5e4-43c6-b4dd-455c0100a176","Type":"ContainerStarted","Data":"6078011c8cd67642f72ff98b1c1a5fdec06f83d5f097fbe4dd12260a7f9f2acc"} Jul 07 18:23:24.082500 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:24.082454 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:24.082826 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:24.082799 2583 patch_prober.go:28] interesting pod/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 07 18:23:24.082888 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:24.082865 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" podUID="a104662b-e5e4-43c6-b4dd-455c0100a176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 07 18:23:25.081992 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:25.081956 2583 patch_prober.go:28] interesting pod/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 07 18:23:25.082486 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:25.082013 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" podUID="a104662b-e5e4-43c6-b4dd-455c0100a176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 07 18:23:25.141784 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:25.141748 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:23:26.082767 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:26.082729 2583 patch_prober.go:28] interesting pod/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 07 18:23:26.083225 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:26.082786 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" podUID="a104662b-e5e4-43c6-b4dd-455c0100a176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 07 18:23:27.082707 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:27.082664 2583 patch_prober.go:28] interesting pod/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 07 18:23:27.082954 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:27.082732 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" podUID="a104662b-e5e4-43c6-b4dd-455c0100a176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 07 18:23:27.626639 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:27.626600 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:27.726494 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:27.726456 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:27.936862 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:27.936771 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs"] Jul 07 18:23:27.937058 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:27.937036 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" podUID="a104662b-e5e4-43c6-b4dd-455c0100a176" containerName="istio-proxy" containerID="cri-o://6078011c8cd67642f72ff98b1c1a5fdec06f83d5f097fbe4dd12260a7f9f2acc" gracePeriod=30 Jul 07 18:23:27.950310 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:27.950281 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:28.185398 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:28.185358 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:28.339086 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:28.339056 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:28.483111 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:28.483074 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:28.663434 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:28.663340 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:33.205218 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.205150 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:33.283752 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.283706 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a104662b-e5e4-43c6-b4dd-455c0100a176-workload-certs\") pod \"a104662b-e5e4-43c6-b4dd-455c0100a176\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " Jul 07 18:23:33.283990 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.283765 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a104662b-e5e4-43c6-b4dd-455c0100a176-credential-socket\") pod \"a104662b-e5e4-43c6-b4dd-455c0100a176\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " Jul 07 18:23:33.283990 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.283792 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kj5f\" (UniqueName: \"kubernetes.io/projected/a104662b-e5e4-43c6-b4dd-455c0100a176-kube-api-access-4kj5f\") pod \"a104662b-e5e4-43c6-b4dd-455c0100a176\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " Jul 07 18:23:33.283990 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.283837 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a104662b-e5e4-43c6-b4dd-455c0100a176-workload-socket\") pod \"a104662b-e5e4-43c6-b4dd-455c0100a176\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " Jul 07 18:23:33.283990 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.283871 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a104662b-e5e4-43c6-b4dd-455c0100a176-istio-envoy\") pod \"a104662b-e5e4-43c6-b4dd-455c0100a176\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " Jul 07 18:23:33.283990 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.283892 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a104662b-e5e4-43c6-b4dd-455c0100a176-istio-token\") pod \"a104662b-e5e4-43c6-b4dd-455c0100a176\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " Jul 07 18:23:33.283990 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.283945 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a104662b-e5e4-43c6-b4dd-455c0100a176-istio-podinfo\") pod \"a104662b-e5e4-43c6-b4dd-455c0100a176\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " Jul 07 18:23:33.283990 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.283990 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a104662b-e5e4-43c6-b4dd-455c0100a176-istio-data\") pod \"a104662b-e5e4-43c6-b4dd-455c0100a176\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " Jul 07 18:23:33.284372 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.284017 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a104662b-e5e4-43c6-b4dd-455c0100a176-istiod-ca-cert\") pod \"a104662b-e5e4-43c6-b4dd-455c0100a176\" (UID: \"a104662b-e5e4-43c6-b4dd-455c0100a176\") " Jul 07 18:23:33.284372 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.284065 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a104662b-e5e4-43c6-b4dd-455c0100a176-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a104662b-e5e4-43c6-b4dd-455c0100a176" (UID: "a104662b-e5e4-43c6-b4dd-455c0100a176"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:23:33.284372 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.284195 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a104662b-e5e4-43c6-b4dd-455c0100a176-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a104662b-e5e4-43c6-b4dd-455c0100a176" (UID: "a104662b-e5e4-43c6-b4dd-455c0100a176"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:23:33.284372 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.284240 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a104662b-e5e4-43c6-b4dd-455c0100a176-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a104662b-e5e4-43c6-b4dd-455c0100a176" (UID: "a104662b-e5e4-43c6-b4dd-455c0100a176"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:23:33.284556 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.284398 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a104662b-e5e4-43c6-b4dd-455c0100a176-workload-socket\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:23:33.284556 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.284417 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a104662b-e5e4-43c6-b4dd-455c0100a176-workload-certs\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:23:33.284556 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.284432 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a104662b-e5e4-43c6-b4dd-455c0100a176-credential-socket\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:23:33.284826 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.284644 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a104662b-e5e4-43c6-b4dd-455c0100a176-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a104662b-e5e4-43c6-b4dd-455c0100a176" (UID: "a104662b-e5e4-43c6-b4dd-455c0100a176"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:23:33.285220 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.285188 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a104662b-e5e4-43c6-b4dd-455c0100a176-istio-data" (OuterVolumeSpecName: "istio-data") pod "a104662b-e5e4-43c6-b4dd-455c0100a176" (UID: "a104662b-e5e4-43c6-b4dd-455c0100a176"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:23:33.286425 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.286395 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a104662b-e5e4-43c6-b4dd-455c0100a176-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a104662b-e5e4-43c6-b4dd-455c0100a176" (UID: "a104662b-e5e4-43c6-b4dd-455c0100a176"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 18:23:33.286539 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.286440 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a104662b-e5e4-43c6-b4dd-455c0100a176-kube-api-access-4kj5f" (OuterVolumeSpecName: "kube-api-access-4kj5f") pod "a104662b-e5e4-43c6-b4dd-455c0100a176" (UID: "a104662b-e5e4-43c6-b4dd-455c0100a176"). InnerVolumeSpecName "kube-api-access-4kj5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:23:33.286797 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.286779 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a104662b-e5e4-43c6-b4dd-455c0100a176-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a104662b-e5e4-43c6-b4dd-455c0100a176" (UID: "a104662b-e5e4-43c6-b4dd-455c0100a176"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:23:33.286837 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.286801 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a104662b-e5e4-43c6-b4dd-455c0100a176-istio-token" (OuterVolumeSpecName: "istio-token") pod "a104662b-e5e4-43c6-b4dd-455c0100a176" (UID: "a104662b-e5e4-43c6-b4dd-455c0100a176"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:23:33.392004 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.391961 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a104662b-e5e4-43c6-b4dd-455c0100a176-istio-envoy\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:23:33.392004 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.391993 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a104662b-e5e4-43c6-b4dd-455c0100a176-istio-token\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:23:33.392004 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.392001 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a104662b-e5e4-43c6-b4dd-455c0100a176-istio-podinfo\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:23:33.392004 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.392009 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a104662b-e5e4-43c6-b4dd-455c0100a176-istio-data\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:23:33.392004 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.392018 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a104662b-e5e4-43c6-b4dd-455c0100a176-istiod-ca-cert\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:23:33.392395 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.392025 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kj5f\" (UniqueName: \"kubernetes.io/projected/a104662b-e5e4-43c6-b4dd-455c0100a176-kube-api-access-4kj5f\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:23:33.953226 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.953190 2583 generic.go:356] "Generic (PLEG): container finished" podID="a104662b-e5e4-43c6-b4dd-455c0100a176" containerID="6078011c8cd67642f72ff98b1c1a5fdec06f83d5f097fbe4dd12260a7f9f2acc" exitCode=0 Jul 07 18:23:33.953411 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.953251 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" Jul 07 18:23:33.953411 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.953275 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" event={"ID":"a104662b-e5e4-43c6-b4dd-455c0100a176","Type":"ContainerDied","Data":"6078011c8cd67642f72ff98b1c1a5fdec06f83d5f097fbe4dd12260a7f9f2acc"} Jul 07 18:23:33.953411 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.953336 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs" event={"ID":"a104662b-e5e4-43c6-b4dd-455c0100a176","Type":"ContainerDied","Data":"ef4ef54aff73063a36a8996c0b0096553a6e58297bd0ed2f7ecb9e2911b7def3"} Jul 07 18:23:33.953411 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.953359 2583 scope.go:117] "RemoveContainer" containerID="6078011c8cd67642f72ff98b1c1a5fdec06f83d5f097fbe4dd12260a7f9f2acc" Jul 07 18:23:33.962161 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.962143 2583 scope.go:117] "RemoveContainer" containerID="6f718f3512d4a01c9f8bbccd97c412184734011c2469d08bef621800297aeb02" Jul 07 18:23:33.970524 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.970499 2583 scope.go:117] "RemoveContainer" containerID="6078011c8cd67642f72ff98b1c1a5fdec06f83d5f097fbe4dd12260a7f9f2acc" Jul 07 18:23:33.970866 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:23:33.970845 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6078011c8cd67642f72ff98b1c1a5fdec06f83d5f097fbe4dd12260a7f9f2acc\": container with ID starting with 6078011c8cd67642f72ff98b1c1a5fdec06f83d5f097fbe4dd12260a7f9f2acc not found: ID does not exist" containerID="6078011c8cd67642f72ff98b1c1a5fdec06f83d5f097fbe4dd12260a7f9f2acc" Jul 07 18:23:33.970936 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.970878 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6078011c8cd67642f72ff98b1c1a5fdec06f83d5f097fbe4dd12260a7f9f2acc"} err="failed to get container status \"6078011c8cd67642f72ff98b1c1a5fdec06f83d5f097fbe4dd12260a7f9f2acc\": rpc error: code = NotFound desc = could not find container \"6078011c8cd67642f72ff98b1c1a5fdec06f83d5f097fbe4dd12260a7f9f2acc\": container with ID starting with 6078011c8cd67642f72ff98b1c1a5fdec06f83d5f097fbe4dd12260a7f9f2acc not found: ID does not exist" Jul 07 18:23:33.970936 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.970919 2583 scope.go:117] "RemoveContainer" containerID="6f718f3512d4a01c9f8bbccd97c412184734011c2469d08bef621800297aeb02" Jul 07 18:23:33.971185 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:23:33.971166 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f718f3512d4a01c9f8bbccd97c412184734011c2469d08bef621800297aeb02\": container with ID starting with 6f718f3512d4a01c9f8bbccd97c412184734011c2469d08bef621800297aeb02 not found: ID does not exist" containerID="6f718f3512d4a01c9f8bbccd97c412184734011c2469d08bef621800297aeb02" Jul 07 18:23:33.971246 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.971196 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f718f3512d4a01c9f8bbccd97c412184734011c2469d08bef621800297aeb02"} err="failed to get container status \"6f718f3512d4a01c9f8bbccd97c412184734011c2469d08bef621800297aeb02\": rpc error: code = NotFound desc = could not find container \"6f718f3512d4a01c9f8bbccd97c412184734011c2469d08bef621800297aeb02\": container with ID starting with 6f718f3512d4a01c9f8bbccd97c412184734011c2469d08bef621800297aeb02 not found: ID does not exist" Jul 07 18:23:33.975408 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.975356 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs"] Jul 07 18:23:33.977578 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:33.977553 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f8b2cd9b-openshift-default-98f9b79cf-g49hs"] Jul 07 18:23:35.429180 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:35.429146 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a104662b-e5e4-43c6-b4dd-455c0100a176" path="/var/lib/kubelet/pods/a104662b-e5e4-43c6-b4dd-455c0100a176/volumes" Jul 07 18:23:35.723975 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:35.723865 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:35.832271 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:35.832227 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:35.942450 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:35.942416 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:36.128817 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:36.128780 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:42.624166 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:42.624126 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:42.734608 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:42.734566 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:49.751607 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:49.751559 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:50.121829 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.121792 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5"] Jul 07 18:23:50.122196 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.122180 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="315ad7a9-9938-47e5-9ede-b3478b8a9ef2" containerName="maas-api" Jul 07 18:23:50.122196 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.122197 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="315ad7a9-9938-47e5-9ede-b3478b8a9ef2" containerName="maas-api" Jul 07 18:23:50.122332 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.122213 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a104662b-e5e4-43c6-b4dd-455c0100a176" containerName="istio-proxy" Jul 07 18:23:50.122332 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.122222 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a104662b-e5e4-43c6-b4dd-455c0100a176" containerName="istio-proxy" Jul 07 18:23:50.122332 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.122238 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a104662b-e5e4-43c6-b4dd-455c0100a176" containerName="istio-proxy" Jul 07 18:23:50.122332 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.122245 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a104662b-e5e4-43c6-b4dd-455c0100a176" containerName="istio-proxy" Jul 07 18:23:50.122477 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.122333 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="315ad7a9-9938-47e5-9ede-b3478b8a9ef2" containerName="maas-api" Jul 07 18:23:50.122477 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.122351 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a104662b-e5e4-43c6-b4dd-455c0100a176" containerName="istio-proxy" Jul 07 18:23:50.122477 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.122359 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a104662b-e5e4-43c6-b4dd-455c0100a176" containerName="istio-proxy" Jul 07 18:23:50.124431 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.124414 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:50.127458 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.127435 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d1f3c1c2-openshift-default-dockercfg-qrn9f\"" Jul 07 18:23:50.144935 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.144885 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5"] Jul 07 18:23:50.158979 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.158928 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:50.252361 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.252314 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5742223b-fb74-4bda-811c-384dd2a5694c-istio-token\") pod \"e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:50.252361 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.252362 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5742223b-fb74-4bda-811c-384dd2a5694c-workload-certs\") pod \"e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:50.252634 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.252446 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5742223b-fb74-4bda-811c-384dd2a5694c-istio-data\") pod \"e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:50.252634 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.252489 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5742223b-fb74-4bda-811c-384dd2a5694c-istio-podinfo\") pod \"e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:50.252634 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.252515 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5742223b-fb74-4bda-811c-384dd2a5694c-istio-envoy\") pod \"e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:50.252634 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.252542 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5742223b-fb74-4bda-811c-384dd2a5694c-workload-socket\") pod \"e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:50.252634 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.252594 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjwkm\" (UniqueName: \"kubernetes.io/projected/5742223b-fb74-4bda-811c-384dd2a5694c-kube-api-access-kjwkm\") pod \"e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:50.252634 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.252630 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5742223b-fb74-4bda-811c-384dd2a5694c-credential-socket\") pod \"e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:50.252875 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.252666 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5742223b-fb74-4bda-811c-384dd2a5694c-istiod-ca-cert\") pod \"e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:50.253141 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.253121 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:50.353620 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.353584 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5742223b-fb74-4bda-811c-384dd2a5694c-workload-socket\") pod \"e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:50.353790 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.353633 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjwkm\" (UniqueName: \"kubernetes.io/projected/5742223b-fb74-4bda-811c-384dd2a5694c-kube-api-access-kjwkm\") pod \"e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:50.353790 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.353756 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5742223b-fb74-4bda-811c-384dd2a5694c-credential-socket\") pod \"e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:50.353870 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.353797 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5742223b-fb74-4bda-811c-384dd2a5694c-istiod-ca-cert\") pod \"e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:50.353870 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.353862 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5742223b-fb74-4bda-811c-384dd2a5694c-istio-token\") pod \"e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:50.354015 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.353890 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5742223b-fb74-4bda-811c-384dd2a5694c-workload-certs\") pod \"e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:50.354015 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.353993 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5742223b-fb74-4bda-811c-384dd2a5694c-istio-data\") pod \"e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:50.354118 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.354016 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5742223b-fb74-4bda-811c-384dd2a5694c-workload-socket\") pod \"e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:50.354118 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.354025 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5742223b-fb74-4bda-811c-384dd2a5694c-istio-podinfo\") pod \"e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:50.354118 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.354082 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5742223b-fb74-4bda-811c-384dd2a5694c-istio-envoy\") pod \"e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:50.354326 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.354207 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5742223b-fb74-4bda-811c-384dd2a5694c-credential-socket\") pod \"e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:50.354326 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.354313 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5742223b-fb74-4bda-811c-384dd2a5694c-workload-certs\") pod \"e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:50.354515 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.354491 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5742223b-fb74-4bda-811c-384dd2a5694c-istio-data\") pod \"e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:50.354608 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.354575 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5742223b-fb74-4bda-811c-384dd2a5694c-istiod-ca-cert\") pod \"e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:50.356793 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.356770 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5742223b-fb74-4bda-811c-384dd2a5694c-istio-envoy\") pod \"e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:50.357384 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.357359 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5742223b-fb74-4bda-811c-384dd2a5694c-istio-podinfo\") pod \"e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:50.367189 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.367160 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5742223b-fb74-4bda-811c-384dd2a5694c-istio-token\") pod \"e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:50.367350 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.367295 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjwkm\" (UniqueName: \"kubernetes.io/projected/5742223b-fb74-4bda-811c-384dd2a5694c-kube-api-access-kjwkm\") pod \"e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:50.374044 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.373972 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:50.435757 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.435721 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:50.581311 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.581204 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5"] Jul 07 18:23:50.583612 ip-10-0-136-192 kubenswrapper[2583]: W0707 18:23:50.583581 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5742223b_fb74_4bda_811c_384dd2a5694c.slice/crio-596605ed8df11f5c3e6c6198f2e0100cfd0c1e0578b83383a3432193bd0be6f5 WatchSource:0}: Error finding container 596605ed8df11f5c3e6c6198f2e0100cfd0c1e0578b83383a3432193bd0be6f5: Status 404 returned error can't find the container with id 596605ed8df11f5c3e6c6198f2e0100cfd0c1e0578b83383a3432193bd0be6f5 Jul 07 18:23:50.585720 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.585685 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:23:50.585806 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.585752 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:23:50.585806 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:50.585781 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:23:51.043011 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:51.042978 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" event={"ID":"5742223b-fb74-4bda-811c-384dd2a5694c","Type":"ContainerStarted","Data":"afefee93e1e46c8214cc32090c8f075af4ba04230f58ae7526021b36afe86e54"} Jul 07 18:23:51.043428 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:51.043029 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" event={"ID":"5742223b-fb74-4bda-811c-384dd2a5694c","Type":"ContainerStarted","Data":"596605ed8df11f5c3e6c6198f2e0100cfd0c1e0578b83383a3432193bd0be6f5"} Jul 07 18:23:51.066309 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:51.066258 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" podStartSLOduration=1.0662398 podStartE2EDuration="1.0662398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:23:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:51.064003362 +0000 UTC m=+1780.146607066" watchObservedRunningTime="2026-07-07 18:23:51.0662398 +0000 UTC m=+1780.148843482" Jul 07 18:23:51.436988 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:51.436889 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:51.438368 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:51.438335 2583 patch_prober.go:28] interesting pod/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 07 18:23:51.438519 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:51.438396 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" podUID="5742223b-fb74-4bda-811c-384dd2a5694c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 07 18:23:52.440656 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:52.440622 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:53.052349 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:53.052317 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:53.053287 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:53.053268 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:23:53.159217 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:53.159185 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:53.305795 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:53.305680 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:53.307396 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:53.307370 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn"] Jul 07 18:23:53.310243 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:53.310220 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn" Jul 07 18:23:53.312708 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:53.312685 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-d1f3c1c2-serving-cert\"" Jul 07 18:23:53.312809 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:53.312765 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-jnv2d\"" Jul 07 18:23:53.326782 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:53.326752 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn"] Jul 07 18:23:53.387112 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:53.387066 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee0caaab-02d4-49c9-b5c9-c9717f82ea09-openshift-service-ca\") pod \"maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn\" (UID: \"ee0caaab-02d4-49c9-b5c9-c9717f82ea09\") " pod="opendatahub/maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn" Jul 07 18:23:53.387309 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:53.387126 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbnck\" (UniqueName: \"kubernetes.io/projected/ee0caaab-02d4-49c9-b5c9-c9717f82ea09-kube-api-access-sbnck\") pod \"maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn\" (UID: \"ee0caaab-02d4-49c9-b5c9-c9717f82ea09\") " pod="opendatahub/maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn" Jul 07 18:23:53.387309 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:53.387182 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee0caaab-02d4-49c9-b5c9-c9717f82ea09-maas-api-tls\") pod \"maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn\" (UID: \"ee0caaab-02d4-49c9-b5c9-c9717f82ea09\") " pod="opendatahub/maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn" Jul 07 18:23:53.398069 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:53.398037 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:53.488556 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:53.488513 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee0caaab-02d4-49c9-b5c9-c9717f82ea09-openshift-service-ca\") pod \"maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn\" (UID: \"ee0caaab-02d4-49c9-b5c9-c9717f82ea09\") " pod="opendatahub/maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn" Jul 07 18:23:53.489065 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:53.488572 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbnck\" (UniqueName: \"kubernetes.io/projected/ee0caaab-02d4-49c9-b5c9-c9717f82ea09-kube-api-access-sbnck\") pod \"maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn\" (UID: \"ee0caaab-02d4-49c9-b5c9-c9717f82ea09\") " pod="opendatahub/maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn" Jul 07 18:23:53.489065 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:53.488616 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee0caaab-02d4-49c9-b5c9-c9717f82ea09-maas-api-tls\") pod \"maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn\" (UID: \"ee0caaab-02d4-49c9-b5c9-c9717f82ea09\") " pod="opendatahub/maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn" Jul 07 18:23:53.489065 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:23:53.488754 2583 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-d1f3c1c2-serving-cert: secret "maas-api-e2e-mt-d1f3c1c2-serving-cert" not found Jul 07 18:23:53.489065 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:23:53.488831 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee0caaab-02d4-49c9-b5c9-c9717f82ea09-maas-api-tls podName:ee0caaab-02d4-49c9-b5c9-c9717f82ea09 nodeName:}" failed. No retries permitted until 2026-07-07 18:23:53.98881506 +0000 UTC m=+1783.071418742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ee0caaab-02d4-49c9-b5c9-c9717f82ea09-maas-api-tls") pod "maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn" (UID: "ee0caaab-02d4-49c9-b5c9-c9717f82ea09") : secret "maas-api-e2e-mt-d1f3c1c2-serving-cert" not found Jul 07 18:23:53.489285 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:53.489266 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee0caaab-02d4-49c9-b5c9-c9717f82ea09-openshift-service-ca\") pod \"maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn\" (UID: \"ee0caaab-02d4-49c9-b5c9-c9717f82ea09\") " pod="opendatahub/maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn" Jul 07 18:23:53.502813 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:53.502785 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbnck\" (UniqueName: \"kubernetes.io/projected/ee0caaab-02d4-49c9-b5c9-c9717f82ea09-kube-api-access-sbnck\") pod \"maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn\" (UID: \"ee0caaab-02d4-49c9-b5c9-c9717f82ea09\") " pod="opendatahub/maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn" Jul 07 18:23:53.978190 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:53.978138 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:53.993630 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:53.993597 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee0caaab-02d4-49c9-b5c9-c9717f82ea09-maas-api-tls\") pod \"maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn\" (UID: \"ee0caaab-02d4-49c9-b5c9-c9717f82ea09\") " pod="opendatahub/maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn" Jul 07 18:23:53.996195 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:53.996168 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee0caaab-02d4-49c9-b5c9-c9717f82ea09-maas-api-tls\") pod \"maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn\" (UID: \"ee0caaab-02d4-49c9-b5c9-c9717f82ea09\") " pod="opendatahub/maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn" Jul 07 18:23:54.157241 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:54.157192 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:54.222994 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:54.222955 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn" Jul 07 18:23:54.257602 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:54.257572 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:54.368578 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:54.368552 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn"] Jul 07 18:23:54.370498 ip-10-0-136-192 kubenswrapper[2583]: W0707 18:23:54.370467 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee0caaab_02d4_49c9_b5c9_c9717f82ea09.slice/crio-62be68bbb01e0b930a8112a4058c0a47847b1261909ac0ec4e4404f6b82bc10c WatchSource:0}: Error finding container 62be68bbb01e0b930a8112a4058c0a47847b1261909ac0ec4e4404f6b82bc10c: Status 404 returned error can't find the container with id 62be68bbb01e0b930a8112a4058c0a47847b1261909ac0ec4e4404f6b82bc10c Jul 07 18:23:55.062361 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:55.062323 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn" event={"ID":"ee0caaab-02d4-49c9-b5c9-c9717f82ea09","Type":"ContainerStarted","Data":"ad07d30b34b92e66c252ebec298e174ec63c8930a3ec116f9a336b536ec4a678"} Jul 07 18:23:55.062799 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:55.062412 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn" Jul 07 18:23:55.062799 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:55.062443 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn" event={"ID":"ee0caaab-02d4-49c9-b5c9-c9717f82ea09","Type":"ContainerStarted","Data":"62be68bbb01e0b930a8112a4058c0a47847b1261909ac0ec4e4404f6b82bc10c"} Jul 07 18:23:55.083055 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:55.083004 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn" podStartSLOduration=1.64174603 podStartE2EDuration="2.082986047s" totalImagesPullingTime="441.240017ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:23:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:23:55.081487098 +0000 UTC m=+1784.164090803" watchObservedRunningTime="2026-07-07 18:23:55.082986047 +0000 UTC m=+1784.165589750" Jul 07 18:23:56.011781 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:56.011745 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:57.586483 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:57.586443 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:23:59.256975 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:23:59.256933 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:24:01.072370 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:01.072340 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn" Jul 07 18:24:08.054052 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:08.054014 2583 patch_prober.go:28] interesting pod/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 07 18:24:08.054475 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:08.054086 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" podUID="5742223b-fb74-4bda-811c-384dd2a5694c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 07 18:24:08.138777 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:08.138745 2583 generic.go:356] "Generic (PLEG): container finished" podID="5742223b-fb74-4bda-811c-384dd2a5694c" containerID="afefee93e1e46c8214cc32090c8f075af4ba04230f58ae7526021b36afe86e54" exitCode=0 Jul 07 18:24:08.138985 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:08.138796 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" event={"ID":"5742223b-fb74-4bda-811c-384dd2a5694c","Type":"ContainerDied","Data":"afefee93e1e46c8214cc32090c8f075af4ba04230f58ae7526021b36afe86e54"} Jul 07 18:24:08.139238 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:08.139222 2583 scope.go:117] "RemoveContainer" containerID="afefee93e1e46c8214cc32090c8f075af4ba04230f58ae7526021b36afe86e54" Jul 07 18:24:08.140221 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:08.140185 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:24:08.140340 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:08.140256 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:24:08.140340 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:08.140287 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:24:09.146201 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:09.146165 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" event={"ID":"5742223b-fb74-4bda-811c-384dd2a5694c","Type":"ContainerStarted","Data":"d81c6a0d7463541e26e4d0541d9b225569b38da67058669453f792c3618b0cbf"} Jul 07 18:24:09.436276 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:09.436186 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:24:09.436432 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:09.436386 2583 patch_prober.go:28] interesting pod/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 07 18:24:09.436474 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:09.436431 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" podUID="5742223b-fb74-4bda-811c-384dd2a5694c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 07 18:24:10.436980 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:10.436936 2583 patch_prober.go:28] interesting pod/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 07 18:24:10.437456 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:10.436997 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" podUID="5742223b-fb74-4bda-811c-384dd2a5694c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 07 18:24:10.735251 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:10.735152 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:11.437044 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:11.437005 2583 patch_prober.go:28] interesting pod/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 07 18:24:11.437432 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:11.437063 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" podUID="5742223b-fb74-4bda-811c-384dd2a5694c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 07 18:24:11.462066 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:11.462040 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wqfm4_997df85c-7632-47b5-a8a6-1c37f3346251/console-operator/2.log" Jul 07 18:24:11.462865 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:11.462842 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wqfm4_997df85c-7632-47b5-a8a6-1c37f3346251/console-operator/2.log" Jul 07 18:24:11.469007 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:11.468985 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zjzpw_5c2d819c-e495-47ec-a1e7-4b2582a195bd/ovn-acl-logging/0.log" Jul 07 18:24:11.469821 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:11.469801 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zjzpw_5c2d819c-e495-47ec-a1e7-4b2582a195bd/ovn-acl-logging/0.log" Jul 07 18:24:12.437175 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:12.437143 2583 patch_prober.go:28] interesting pod/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 07 18:24:12.437624 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:12.437212 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" podUID="5742223b-fb74-4bda-811c-384dd2a5694c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 07 18:24:13.440990 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:13.440960 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:24:13.441432 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:13.441336 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:24:13.442844 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:13.442820 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:24:13.645726 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:13.645690 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:24:29.168556 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:29.168516 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:24:30.161844 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:30.161808 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:30.728920 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:30.728864 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn"] Jul 07 18:24:30.729437 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:30.729185 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn" podUID="ee0caaab-02d4-49c9-b5c9-c9717f82ea09" containerName="maas-api" containerID="cri-o://ad07d30b34b92e66c252ebec298e174ec63c8930a3ec116f9a336b536ec4a678" gracePeriod=30 Jul 07 18:24:30.776339 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:30.776298 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:30.996896 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:30.996870 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn" Jul 07 18:24:31.154867 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:31.154826 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbnck\" (UniqueName: \"kubernetes.io/projected/ee0caaab-02d4-49c9-b5c9-c9717f82ea09-kube-api-access-sbnck\") pod \"ee0caaab-02d4-49c9-b5c9-c9717f82ea09\" (UID: \"ee0caaab-02d4-49c9-b5c9-c9717f82ea09\") " Jul 07 18:24:31.154867 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:31.154874 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee0caaab-02d4-49c9-b5c9-c9717f82ea09-openshift-service-ca\") pod \"ee0caaab-02d4-49c9-b5c9-c9717f82ea09\" (UID: \"ee0caaab-02d4-49c9-b5c9-c9717f82ea09\") " Jul 07 18:24:31.155159 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:31.154897 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee0caaab-02d4-49c9-b5c9-c9717f82ea09-maas-api-tls\") pod \"ee0caaab-02d4-49c9-b5c9-c9717f82ea09\" (UID: \"ee0caaab-02d4-49c9-b5c9-c9717f82ea09\") " Jul 07 18:24:31.155270 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:31.155243 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee0caaab-02d4-49c9-b5c9-c9717f82ea09-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ee0caaab-02d4-49c9-b5c9-c9717f82ea09" (UID: "ee0caaab-02d4-49c9-b5c9-c9717f82ea09"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:24:31.157323 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:31.157297 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee0caaab-02d4-49c9-b5c9-c9717f82ea09-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ee0caaab-02d4-49c9-b5c9-c9717f82ea09" (UID: "ee0caaab-02d4-49c9-b5c9-c9717f82ea09"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:24:31.157423 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:31.157366 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee0caaab-02d4-49c9-b5c9-c9717f82ea09-kube-api-access-sbnck" (OuterVolumeSpecName: "kube-api-access-sbnck") pod "ee0caaab-02d4-49c9-b5c9-c9717f82ea09" (UID: "ee0caaab-02d4-49c9-b5c9-c9717f82ea09"). InnerVolumeSpecName "kube-api-access-sbnck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:24:31.256715 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:31.256622 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbnck\" (UniqueName: \"kubernetes.io/projected/ee0caaab-02d4-49c9-b5c9-c9717f82ea09-kube-api-access-sbnck\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:24:31.256715 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:31.256653 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee0caaab-02d4-49c9-b5c9-c9717f82ea09-openshift-service-ca\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:24:31.256715 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:31.256667 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee0caaab-02d4-49c9-b5c9-c9717f82ea09-maas-api-tls\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:24:31.257512 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:31.257490 2583 generic.go:356] "Generic (PLEG): container finished" podID="ee0caaab-02d4-49c9-b5c9-c9717f82ea09" containerID="ad07d30b34b92e66c252ebec298e174ec63c8930a3ec116f9a336b536ec4a678" exitCode=0 Jul 07 18:24:31.257571 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:31.257531 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn" Jul 07 18:24:31.257641 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:31.257529 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn" event={"ID":"ee0caaab-02d4-49c9-b5c9-c9717f82ea09","Type":"ContainerDied","Data":"ad07d30b34b92e66c252ebec298e174ec63c8930a3ec116f9a336b536ec4a678"} Jul 07 18:24:31.257750 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:31.257731 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn" event={"ID":"ee0caaab-02d4-49c9-b5c9-c9717f82ea09","Type":"ContainerDied","Data":"62be68bbb01e0b930a8112a4058c0a47847b1261909ac0ec4e4404f6b82bc10c"} Jul 07 18:24:31.257814 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:31.257774 2583 scope.go:117] "RemoveContainer" containerID="ad07d30b34b92e66c252ebec298e174ec63c8930a3ec116f9a336b536ec4a678" Jul 07 18:24:31.268143 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:31.268118 2583 scope.go:117] "RemoveContainer" containerID="ad07d30b34b92e66c252ebec298e174ec63c8930a3ec116f9a336b536ec4a678" Jul 07 18:24:31.268438 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:24:31.268419 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad07d30b34b92e66c252ebec298e174ec63c8930a3ec116f9a336b536ec4a678\": container with ID starting with ad07d30b34b92e66c252ebec298e174ec63c8930a3ec116f9a336b536ec4a678 not found: ID does not exist" containerID="ad07d30b34b92e66c252ebec298e174ec63c8930a3ec116f9a336b536ec4a678" Jul 07 18:24:31.268517 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:31.268444 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad07d30b34b92e66c252ebec298e174ec63c8930a3ec116f9a336b536ec4a678"} err="failed to get container status \"ad07d30b34b92e66c252ebec298e174ec63c8930a3ec116f9a336b536ec4a678\": rpc error: code = NotFound desc = could not find container \"ad07d30b34b92e66c252ebec298e174ec63c8930a3ec116f9a336b536ec4a678\": container with ID starting with ad07d30b34b92e66c252ebec298e174ec63c8930a3ec116f9a336b536ec4a678 not found: ID does not exist" Jul 07 18:24:31.284224 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:31.284191 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn"] Jul 07 18:24:31.288571 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:31.288538 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-d1f3c1c2-9c9bf7c76-9fvxn"] Jul 07 18:24:31.429494 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:31.429457 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee0caaab-02d4-49c9-b5c9-c9717f82ea09" path="/var/lib/kubelet/pods/ee0caaab-02d4-49c9-b5c9-c9717f82ea09/volumes" Jul 07 18:24:36.321178 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:36.321142 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:24:41.253651 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:41.253612 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:24:41.394644 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:24:41.394607 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:25:17.808352 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:17.808322 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:25:19.697855 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:19.697822 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:25:20.848770 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:20.848731 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:25:25.624178 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:25.624142 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:25:27.026722 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:27.026687 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:25:27.129215 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:27.129178 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:25:27.316988 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:27.316951 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5"] Jul 07 18:25:27.317223 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:27.317197 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" podUID="5742223b-fb74-4bda-811c-384dd2a5694c" containerName="istio-proxy" containerID="cri-o://d81c6a0d7463541e26e4d0541d9b225569b38da67058669453f792c3618b0cbf" gracePeriod=30 Jul 07 18:25:27.340662 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:27.340619 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:25:27.596776 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:27.596677 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:25:28.444338 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:28.444303 2583 patch_prober.go:28] interesting pod/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 18:25:28.444716 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:28.444363 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" podUID="5742223b-fb74-4bda-811c-384dd2a5694c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 18:25:28.813669 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:28.813629 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:25:28.914546 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:28.914508 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:25:29.021212 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:29.021178 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:25:29.963556 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:29.963525 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:25:30.101069 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:30.101022 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:25:31.640490 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:31.640399 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:25:32.553501 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.553467 2583 generic.go:356] "Generic (PLEG): container finished" podID="5742223b-fb74-4bda-811c-384dd2a5694c" containerID="d81c6a0d7463541e26e4d0541d9b225569b38da67058669453f792c3618b0cbf" exitCode=0 Jul 07 18:25:32.553703 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.553535 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" event={"ID":"5742223b-fb74-4bda-811c-384dd2a5694c","Type":"ContainerDied","Data":"d81c6a0d7463541e26e4d0541d9b225569b38da67058669453f792c3618b0cbf"} Jul 07 18:25:32.553703 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.553610 2583 scope.go:117] "RemoveContainer" containerID="afefee93e1e46c8214cc32090c8f075af4ba04230f58ae7526021b36afe86e54" Jul 07 18:25:32.586015 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.585949 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:25:32.653431 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.653395 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5742223b-fb74-4bda-811c-384dd2a5694c-istio-data\") pod \"5742223b-fb74-4bda-811c-384dd2a5694c\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " Jul 07 18:25:32.653954 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.653441 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5742223b-fb74-4bda-811c-384dd2a5694c-workload-certs\") pod \"5742223b-fb74-4bda-811c-384dd2a5694c\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " Jul 07 18:25:32.653954 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.653468 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjwkm\" (UniqueName: \"kubernetes.io/projected/5742223b-fb74-4bda-811c-384dd2a5694c-kube-api-access-kjwkm\") pod \"5742223b-fb74-4bda-811c-384dd2a5694c\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " Jul 07 18:25:32.653954 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.653510 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5742223b-fb74-4bda-811c-384dd2a5694c-credential-socket\") pod \"5742223b-fb74-4bda-811c-384dd2a5694c\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " Jul 07 18:25:32.653954 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.653552 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5742223b-fb74-4bda-811c-384dd2a5694c-istiod-ca-cert\") pod \"5742223b-fb74-4bda-811c-384dd2a5694c\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " Jul 07 18:25:32.653954 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.653685 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5742223b-fb74-4bda-811c-384dd2a5694c-istio-podinfo\") pod \"5742223b-fb74-4bda-811c-384dd2a5694c\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " Jul 07 18:25:32.653954 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.653771 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5742223b-fb74-4bda-811c-384dd2a5694c-istio-envoy\") pod \"5742223b-fb74-4bda-811c-384dd2a5694c\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " Jul 07 18:25:32.653954 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.653791 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5742223b-fb74-4bda-811c-384dd2a5694c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5742223b-fb74-4bda-811c-384dd2a5694c" (UID: "5742223b-fb74-4bda-811c-384dd2a5694c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:25:32.653954 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.653819 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5742223b-fb74-4bda-811c-384dd2a5694c-workload-socket\") pod \"5742223b-fb74-4bda-811c-384dd2a5694c\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " Jul 07 18:25:32.653954 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.653856 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5742223b-fb74-4bda-811c-384dd2a5694c-istio-token\") pod \"5742223b-fb74-4bda-811c-384dd2a5694c\" (UID: \"5742223b-fb74-4bda-811c-384dd2a5694c\") " Jul 07 18:25:32.653954 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.653929 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5742223b-fb74-4bda-811c-384dd2a5694c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5742223b-fb74-4bda-811c-384dd2a5694c" (UID: "5742223b-fb74-4bda-811c-384dd2a5694c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:25:32.654467 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.654040 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5742223b-fb74-4bda-811c-384dd2a5694c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5742223b-fb74-4bda-811c-384dd2a5694c" (UID: "5742223b-fb74-4bda-811c-384dd2a5694c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:25:32.654467 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.654197 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5742223b-fb74-4bda-811c-384dd2a5694c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5742223b-fb74-4bda-811c-384dd2a5694c" (UID: "5742223b-fb74-4bda-811c-384dd2a5694c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:25:32.654467 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.654426 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5742223b-fb74-4bda-811c-384dd2a5694c-istio-data" (OuterVolumeSpecName: "istio-data") pod "5742223b-fb74-4bda-811c-384dd2a5694c" (UID: "5742223b-fb74-4bda-811c-384dd2a5694c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:25:32.654587 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.654573 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5742223b-fb74-4bda-811c-384dd2a5694c-workload-socket\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:25:32.654631 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.654593 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5742223b-fb74-4bda-811c-384dd2a5694c-workload-certs\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:25:32.654631 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.654610 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5742223b-fb74-4bda-811c-384dd2a5694c-credential-socket\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:25:32.654631 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.654624 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5742223b-fb74-4bda-811c-384dd2a5694c-istiod-ca-cert\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:25:32.656049 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.656010 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5742223b-fb74-4bda-811c-384dd2a5694c-kube-api-access-kjwkm" (OuterVolumeSpecName: "kube-api-access-kjwkm") pod "5742223b-fb74-4bda-811c-384dd2a5694c" (UID: "5742223b-fb74-4bda-811c-384dd2a5694c"). InnerVolumeSpecName "kube-api-access-kjwkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:25:32.656205 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.656184 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5742223b-fb74-4bda-811c-384dd2a5694c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5742223b-fb74-4bda-811c-384dd2a5694c" (UID: "5742223b-fb74-4bda-811c-384dd2a5694c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 18:25:32.656500 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.656481 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5742223b-fb74-4bda-811c-384dd2a5694c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5742223b-fb74-4bda-811c-384dd2a5694c" (UID: "5742223b-fb74-4bda-811c-384dd2a5694c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:25:32.656650 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.656629 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5742223b-fb74-4bda-811c-384dd2a5694c-istio-token" (OuterVolumeSpecName: "istio-token") pod "5742223b-fb74-4bda-811c-384dd2a5694c" (UID: "5742223b-fb74-4bda-811c-384dd2a5694c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:25:32.673173 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.673139 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:25:32.755795 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.755761 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjwkm\" (UniqueName: \"kubernetes.io/projected/5742223b-fb74-4bda-811c-384dd2a5694c-kube-api-access-kjwkm\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:25:32.755795 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.755792 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5742223b-fb74-4bda-811c-384dd2a5694c-istio-podinfo\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:25:32.755795 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.755801 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5742223b-fb74-4bda-811c-384dd2a5694c-istio-envoy\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:25:32.756072 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.755809 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5742223b-fb74-4bda-811c-384dd2a5694c-istio-token\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:25:32.756072 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:32.755816 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5742223b-fb74-4bda-811c-384dd2a5694c-istio-data\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:25:33.559323 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:33.559293 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" event={"ID":"5742223b-fb74-4bda-811c-384dd2a5694c","Type":"ContainerDied","Data":"596605ed8df11f5c3e6c6198f2e0100cfd0c1e0578b83383a3432193bd0be6f5"} Jul 07 18:25:33.559497 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:33.559352 2583 scope.go:117] "RemoveContainer" containerID="d81c6a0d7463541e26e4d0541d9b225569b38da67058669453f792c3618b0cbf" Jul 07 18:25:33.559497 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:33.559304 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5" Jul 07 18:25:33.591138 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:33.591102 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5"] Jul 07 18:25:33.618068 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:33.618021 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d1f3c1c2-openshift-default-5bc7dfb759-d4bh5"] Jul 07 18:25:35.429349 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:35.429314 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5742223b-fb74-4bda-811c-384dd2a5694c" path="/var/lib/kubelet/pods/5742223b-fb74-4bda-811c-384dd2a5694c/volumes" Jul 07 18:25:38.520880 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:38.520847 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:25:38.616721 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:38.611598 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:25:39.313930 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:39.313858 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:25:45.966231 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:45.966200 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:25:46.267241 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:46.267115 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:25:46.384118 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:46.384088 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:25:56.865050 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:56.865020 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:25:57.300204 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:25:57.300173 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:18.302345 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:18.302305 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:24.242247 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:24.242211 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:24.763826 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:24.763789 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:24.835680 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:24.835644 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:25.135073 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.135036 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:25.441093 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.440974 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2"] Jul 07 18:26:25.441486 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.441365 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5742223b-fb74-4bda-811c-384dd2a5694c" containerName="istio-proxy" Jul 07 18:26:25.441486 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.441380 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5742223b-fb74-4bda-811c-384dd2a5694c" containerName="istio-proxy" Jul 07 18:26:25.441486 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.441394 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5742223b-fb74-4bda-811c-384dd2a5694c" containerName="istio-proxy" Jul 07 18:26:25.441486 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.441400 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5742223b-fb74-4bda-811c-384dd2a5694c" containerName="istio-proxy" Jul 07 18:26:25.441486 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.441419 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee0caaab-02d4-49c9-b5c9-c9717f82ea09" containerName="maas-api" Jul 07 18:26:25.441486 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.441424 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee0caaab-02d4-49c9-b5c9-c9717f82ea09" containerName="maas-api" Jul 07 18:26:25.441709 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.441498 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee0caaab-02d4-49c9-b5c9-c9717f82ea09" containerName="maas-api" Jul 07 18:26:25.441709 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.441506 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5742223b-fb74-4bda-811c-384dd2a5694c" containerName="istio-proxy" Jul 07 18:26:25.443484 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.443464 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:25.446098 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.446075 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-95ab96-openshift-default-dockercfg-rw9nz\"" Jul 07 18:26:25.459940 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.459889 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2"] Jul 07 18:26:25.469801 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.469766 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:25.547048 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.547014 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-istio-podinfo\") pod \"e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:25.547229 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.547096 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-258t9\" (UniqueName: \"kubernetes.io/projected/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-kube-api-access-258t9\") pod \"e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:25.547229 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.547135 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-istio-data\") pod \"e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:25.547229 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.547173 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-workload-socket\") pod \"e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:25.547229 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.547213 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-istio-envoy\") pod \"e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:25.547415 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.547254 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-workload-certs\") pod \"e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:25.547878 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.547551 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-istio-token\") pod \"e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:25.547878 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.547665 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-istiod-ca-cert\") pod \"e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:25.547878 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.547744 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-credential-socket\") pod \"e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:25.648606 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.648569 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-istio-podinfo\") pod \"e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:25.648804 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.648627 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-258t9\" (UniqueName: \"kubernetes.io/projected/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-kube-api-access-258t9\") pod \"e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:25.648804 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.648649 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-istio-data\") pod \"e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:25.648804 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.648675 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-workload-socket\") pod \"e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:25.648804 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.648699 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-istio-envoy\") pod \"e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:25.649051 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.648916 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-workload-certs\") pod \"e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:25.649051 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.648985 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-istio-token\") pod \"e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:25.649051 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.649014 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-istiod-ca-cert\") pod \"e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:25.649051 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.649045 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-credential-socket\") pod \"e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:25.649255 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.649063 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-istio-data\") pod \"e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:25.649255 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.649151 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-workload-socket\") pod \"e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:25.649255 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.649243 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-workload-certs\") pod \"e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:25.649508 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.649446 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-credential-socket\") pod \"e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:25.649717 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.649698 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-istiod-ca-cert\") pod \"e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:25.651089 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.651069 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-istio-envoy\") pod \"e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:25.651423 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.651400 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-istio-podinfo\") pod \"e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:25.660180 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.660153 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-258t9\" (UniqueName: \"kubernetes.io/projected/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-kube-api-access-258t9\") pod \"e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:25.660180 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.660167 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-istio-token\") pod \"e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:25.668074 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.668047 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:25.753298 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.753210 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:25.896071 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.896043 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2"] Jul 07 18:26:25.898511 ip-10-0-136-192 kubenswrapper[2583]: W0707 18:26:25.898485 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42cc52ec_3a4e_4507_b417_7f2013c6c3b1.slice/crio-aa318875cebf3b2e563683dcf5e827160cd664a91a9999a79e697b6cf06d10b2 WatchSource:0}: Error finding container aa318875cebf3b2e563683dcf5e827160cd664a91a9999a79e697b6cf06d10b2: Status 404 returned error can't find the container with id aa318875cebf3b2e563683dcf5e827160cd664a91a9999a79e697b6cf06d10b2 Jul 07 18:26:25.900604 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.900569 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:26:25.900675 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.900649 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:26:25.900713 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:25.900676 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:26:26.810007 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:26.809966 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" event={"ID":"42cc52ec-3a4e-4507-b417-7f2013c6c3b1","Type":"ContainerStarted","Data":"0f982fdff8907ad6dc78b0594de90fb88a8fbf7be274abc18c814cc40264b6e4"} Jul 07 18:26:26.810409 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:26.810049 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" event={"ID":"42cc52ec-3a4e-4507-b417-7f2013c6c3b1","Type":"ContainerStarted","Data":"aa318875cebf3b2e563683dcf5e827160cd664a91a9999a79e697b6cf06d10b2"} Jul 07 18:26:26.832758 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:26.832704 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" podStartSLOduration=1.832686179 podStartE2EDuration="1.832686179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:26:26.830338335 +0000 UTC m=+1935.912942039" watchObservedRunningTime="2026-07-07 18:26:26.832686179 +0000 UTC m=+1935.915289881" Jul 07 18:26:27.754338 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:27.754288 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:27.759078 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:27.759046 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:27.814486 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:27.814456 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:27.815398 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:27.815375 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:28.841777 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:28.841744 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:28.945027 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:28.944990 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:28.960949 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:28.960914 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws"] Jul 07 18:26:28.961567 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:28.961533 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5742223b-fb74-4bda-811c-384dd2a5694c" containerName="istio-proxy" Jul 07 18:26:28.963701 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:28.963681 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws" Jul 07 18:26:28.966182 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:28.966162 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-jnv2d\"" Jul 07 18:26:28.966627 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:28.966611 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-95ab96-serving-cert\"" Jul 07 18:26:28.978469 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:28.978442 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws"] Jul 07 18:26:29.043046 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:29.043013 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:29.086375 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:29.086320 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6x98\" (UniqueName: \"kubernetes.io/projected/86105961-ca05-438b-a080-607318a3b94c-kube-api-access-r6x98\") pod \"maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws\" (UID: \"86105961-ca05-438b-a080-607318a3b94c\") " pod="opendatahub/maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws" Jul 07 18:26:29.086375 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:29.086384 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86105961-ca05-438b-a080-607318a3b94c-maas-api-tls\") pod \"maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws\" (UID: \"86105961-ca05-438b-a080-607318a3b94c\") " pod="opendatahub/maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws" Jul 07 18:26:29.086599 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:29.086403 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86105961-ca05-438b-a080-607318a3b94c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws\" (UID: \"86105961-ca05-438b-a080-607318a3b94c\") " pod="opendatahub/maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws" Jul 07 18:26:29.187715 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:29.187623 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6x98\" (UniqueName: \"kubernetes.io/projected/86105961-ca05-438b-a080-607318a3b94c-kube-api-access-r6x98\") pod \"maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws\" (UID: \"86105961-ca05-438b-a080-607318a3b94c\") " pod="opendatahub/maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws" Jul 07 18:26:29.187715 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:29.187671 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86105961-ca05-438b-a080-607318a3b94c-maas-api-tls\") pod \"maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws\" (UID: \"86105961-ca05-438b-a080-607318a3b94c\") " pod="opendatahub/maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws" Jul 07 18:26:29.187715 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:29.187697 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86105961-ca05-438b-a080-607318a3b94c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws\" (UID: \"86105961-ca05-438b-a080-607318a3b94c\") " pod="opendatahub/maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws" Jul 07 18:26:29.187988 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:26:29.187833 2583 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-a-95ab96-serving-cert: secret "maas-api-e2e-inf-a-95ab96-serving-cert" not found Jul 07 18:26:29.187988 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:26:29.187938 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86105961-ca05-438b-a080-607318a3b94c-maas-api-tls podName:86105961-ca05-438b-a080-607318a3b94c nodeName:}" failed. No retries permitted until 2026-07-07 18:26:29.687885673 +0000 UTC m=+1938.770489355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/86105961-ca05-438b-a080-607318a3b94c-maas-api-tls") pod "maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws" (UID: "86105961-ca05-438b-a080-607318a3b94c") : secret "maas-api-e2e-inf-a-95ab96-serving-cert" not found Jul 07 18:26:29.188386 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:29.188369 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86105961-ca05-438b-a080-607318a3b94c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws\" (UID: \"86105961-ca05-438b-a080-607318a3b94c\") " pod="opendatahub/maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws" Jul 07 18:26:29.200440 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:29.200399 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6x98\" (UniqueName: \"kubernetes.io/projected/86105961-ca05-438b-a080-607318a3b94c-kube-api-access-r6x98\") pod \"maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws\" (UID: \"86105961-ca05-438b-a080-607318a3b94c\") " pod="opendatahub/maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws" Jul 07 18:26:29.512571 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:29.512485 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:29.613951 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:29.613887 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:29.693629 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:29.693566 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86105961-ca05-438b-a080-607318a3b94c-maas-api-tls\") pod \"maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws\" (UID: \"86105961-ca05-438b-a080-607318a3b94c\") " pod="opendatahub/maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws" Jul 07 18:26:29.696221 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:29.696195 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86105961-ca05-438b-a080-607318a3b94c-maas-api-tls\") pod \"maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws\" (UID: \"86105961-ca05-438b-a080-607318a3b94c\") " pod="opendatahub/maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws" Jul 07 18:26:29.724337 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:29.724303 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:29.876579 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:29.876546 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws" Jul 07 18:26:30.027463 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:30.027436 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws"] Jul 07 18:26:30.029445 ip-10-0-136-192 kubenswrapper[2583]: W0707 18:26:30.029417 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86105961_ca05_438b_a080_607318a3b94c.slice/crio-092dc65e41eca7564a7bd68ebc11f76bd993374a323a7ed36e8e06ac5bdb6ba7 WatchSource:0}: Error finding container 092dc65e41eca7564a7bd68ebc11f76bd993374a323a7ed36e8e06ac5bdb6ba7: Status 404 returned error can't find the container with id 092dc65e41eca7564a7bd68ebc11f76bd993374a323a7ed36e8e06ac5bdb6ba7 Jul 07 18:26:30.030865 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:30.030845 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 18:26:30.830134 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:30.830092 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws" event={"ID":"86105961-ca05-438b-a080-607318a3b94c","Type":"ContainerStarted","Data":"6b297e4dcfc331220eb8acdc37e6ca999482d93474f5372367f7d2f33a31030a"} Jul 07 18:26:30.830322 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:30.830151 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws" event={"ID":"86105961-ca05-438b-a080-607318a3b94c","Type":"ContainerStarted","Data":"092dc65e41eca7564a7bd68ebc11f76bd993374a323a7ed36e8e06ac5bdb6ba7"} Jul 07 18:26:30.830322 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:30.830195 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws" Jul 07 18:26:30.851511 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:30.851459 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws" podStartSLOduration=2.208736118 podStartE2EDuration="2.851440469s" totalImagesPullingTime="642.704351ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:26:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:26:30.849842257 +0000 UTC m=+1939.932445960" watchObservedRunningTime="2026-07-07 18:26:30.851440469 +0000 UTC m=+1939.934044210" Jul 07 18:26:33.334433 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:33.334400 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:33.713311 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:33.713217 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:34.007776 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.007678 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8"] Jul 07 18:26:34.010413 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.010387 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:34.013286 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.013260 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-46d729-openshift-default-dockercfg-c6bgp\"" Jul 07 18:26:34.026557 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.026523 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8"] Jul 07 18:26:34.037735 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.037702 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d414c637-7f80-4112-bf4c-9e0aefb24133-istio-envoy\") pod \"e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:34.037948 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.037761 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d414c637-7f80-4112-bf4c-9e0aefb24133-credential-socket\") pod \"e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:34.037948 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.037796 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d414c637-7f80-4112-bf4c-9e0aefb24133-istio-podinfo\") pod \"e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:34.037948 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.037829 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d414c637-7f80-4112-bf4c-9e0aefb24133-istiod-ca-cert\") pod \"e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:34.037948 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.037851 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d414c637-7f80-4112-bf4c-9e0aefb24133-istio-data\") pod \"e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:34.038115 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.037949 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d414c637-7f80-4112-bf4c-9e0aefb24133-workload-socket\") pod \"e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:34.038115 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.037975 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj47m\" (UniqueName: \"kubernetes.io/projected/d414c637-7f80-4112-bf4c-9e0aefb24133-kube-api-access-mj47m\") pod \"e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:34.038115 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.038011 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d414c637-7f80-4112-bf4c-9e0aefb24133-workload-certs\") pod \"e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:34.038115 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.038072 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d414c637-7f80-4112-bf4c-9e0aefb24133-istio-token\") pod \"e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:34.042442 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.042414 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:34.115467 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.115431 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:34.139422 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.139379 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d414c637-7f80-4112-bf4c-9e0aefb24133-credential-socket\") pod \"e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:34.139626 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.139436 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d414c637-7f80-4112-bf4c-9e0aefb24133-istio-podinfo\") pod \"e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:34.139626 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.139479 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d414c637-7f80-4112-bf4c-9e0aefb24133-istiod-ca-cert\") pod \"e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:34.139626 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.139513 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d414c637-7f80-4112-bf4c-9e0aefb24133-istio-data\") pod \"e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:34.139790 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.139659 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d414c637-7f80-4112-bf4c-9e0aefb24133-workload-socket\") pod \"e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:34.139790 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.139697 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj47m\" (UniqueName: \"kubernetes.io/projected/d414c637-7f80-4112-bf4c-9e0aefb24133-kube-api-access-mj47m\") pod \"e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:34.139790 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.139752 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d414c637-7f80-4112-bf4c-9e0aefb24133-workload-certs\") pod \"e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:34.139790 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.139781 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d414c637-7f80-4112-bf4c-9e0aefb24133-istio-token\") pod \"e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:34.140028 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.139848 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d414c637-7f80-4112-bf4c-9e0aefb24133-credential-socket\") pod \"e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:34.140028 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.139857 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d414c637-7f80-4112-bf4c-9e0aefb24133-istio-envoy\") pod \"e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:34.140028 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.139990 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d414c637-7f80-4112-bf4c-9e0aefb24133-istio-data\") pod \"e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:34.140182 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.140148 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d414c637-7f80-4112-bf4c-9e0aefb24133-workload-socket\") pod \"e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:34.140258 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.140225 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d414c637-7f80-4112-bf4c-9e0aefb24133-workload-certs\") pod \"e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:34.140607 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.140586 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d414c637-7f80-4112-bf4c-9e0aefb24133-istiod-ca-cert\") pod \"e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:34.142159 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.142131 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d414c637-7f80-4112-bf4c-9e0aefb24133-istio-envoy\") pod \"e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:34.142270 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.142170 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d414c637-7f80-4112-bf4c-9e0aefb24133-istio-podinfo\") pod \"e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:34.148773 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.148744 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d414c637-7f80-4112-bf4c-9e0aefb24133-istio-token\") pod \"e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:34.148873 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.148853 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj47m\" (UniqueName: \"kubernetes.io/projected/d414c637-7f80-4112-bf4c-9e0aefb24133-kube-api-access-mj47m\") pod \"e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:34.321830 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.321790 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:34.466225 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.466196 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8"] Jul 07 18:26:34.468496 ip-10-0-136-192 kubenswrapper[2583]: W0707 18:26:34.468469 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd414c637_7f80_4112_bf4c_9e0aefb24133.slice/crio-ace1ffec2f0d01454dc664b40321545979f1d393d48d56245b64fb8eb4903cd3 WatchSource:0}: Error finding container ace1ffec2f0d01454dc664b40321545979f1d393d48d56245b64fb8eb4903cd3: Status 404 returned error can't find the container with id ace1ffec2f0d01454dc664b40321545979f1d393d48d56245b64fb8eb4903cd3 Jul 07 18:26:34.470887 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.470853 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:26:34.470986 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.470959 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:26:34.471024 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.470993 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:26:34.747283 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.747183 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:26:34.849764 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.849727 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" event={"ID":"d414c637-7f80-4112-bf4c-9e0aefb24133","Type":"ContainerStarted","Data":"21f27cdc9e3ddf06e4ae35d0fd2fa7ac1a27a419997c6b00051bfb29146d8431"} Jul 07 18:26:34.849987 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.849784 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" event={"ID":"d414c637-7f80-4112-bf4c-9e0aefb24133","Type":"ContainerStarted","Data":"ace1ffec2f0d01454dc664b40321545979f1d393d48d56245b64fb8eb4903cd3"} Jul 07 18:26:34.871529 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:34.871482 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" podStartSLOduration=1.871460409 podStartE2EDuration="1.871460409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:26:34.870961937 +0000 UTC m=+1943.953565640" watchObservedRunningTime="2026-07-07 18:26:34.871460409 +0000 UTC m=+1943.954064111" Jul 07 18:26:35.322524 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:35.322482 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:35.327264 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:35.327237 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:35.855264 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:35.855229 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:35.856371 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:35.856353 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:36.840551 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:36.840519 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws" Jul 07 18:26:36.911963 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:36.911877 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:37.010766 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:37.010734 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:37.078865 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:37.078825 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx"] Jul 07 18:26:37.081503 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:37.081483 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx" Jul 07 18:26:37.083985 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:37.083956 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-46d729-serving-cert\"" Jul 07 18:26:37.095646 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:37.095611 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx"] Jul 07 18:26:37.170644 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:37.170605 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4e32b23-6587-494c-8e35-0766819be2de-openshift-service-ca\") pod \"maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx\" (UID: \"a4e32b23-6587-494c-8e35-0766819be2de\") " pod="opendatahub/maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx" Jul 07 18:26:37.170851 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:37.170690 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4e32b23-6587-494c-8e35-0766819be2de-maas-api-tls\") pod \"maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx\" (UID: \"a4e32b23-6587-494c-8e35-0766819be2de\") " pod="opendatahub/maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx" Jul 07 18:26:37.170851 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:37.170750 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5mxh\" (UniqueName: \"kubernetes.io/projected/a4e32b23-6587-494c-8e35-0766819be2de-kube-api-access-n5mxh\") pod \"maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx\" (UID: \"a4e32b23-6587-494c-8e35-0766819be2de\") " pod="opendatahub/maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx" Jul 07 18:26:37.173254 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:37.173229 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:37.271922 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:37.271880 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4e32b23-6587-494c-8e35-0766819be2de-maas-api-tls\") pod \"maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx\" (UID: \"a4e32b23-6587-494c-8e35-0766819be2de\") " pod="opendatahub/maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx" Jul 07 18:26:37.272114 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:37.271979 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5mxh\" (UniqueName: \"kubernetes.io/projected/a4e32b23-6587-494c-8e35-0766819be2de-kube-api-access-n5mxh\") pod \"maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx\" (UID: \"a4e32b23-6587-494c-8e35-0766819be2de\") " pod="opendatahub/maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx" Jul 07 18:26:37.272114 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:26:37.272044 2583 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-b-46d729-serving-cert: secret "maas-api-e2e-inf-b-46d729-serving-cert" not found Jul 07 18:26:37.272114 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:37.272059 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4e32b23-6587-494c-8e35-0766819be2de-openshift-service-ca\") pod \"maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx\" (UID: \"a4e32b23-6587-494c-8e35-0766819be2de\") " pod="opendatahub/maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx" Jul 07 18:26:37.272272 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:26:37.272124 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4e32b23-6587-494c-8e35-0766819be2de-maas-api-tls podName:a4e32b23-6587-494c-8e35-0766819be2de nodeName:}" failed. No retries permitted until 2026-07-07 18:26:37.772102481 +0000 UTC m=+1946.854706166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a4e32b23-6587-494c-8e35-0766819be2de-maas-api-tls") pod "maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx" (UID: "a4e32b23-6587-494c-8e35-0766819be2de") : secret "maas-api-e2e-inf-b-46d729-serving-cert" not found Jul 07 18:26:37.272663 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:37.272642 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4e32b23-6587-494c-8e35-0766819be2de-openshift-service-ca\") pod \"maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx\" (UID: \"a4e32b23-6587-494c-8e35-0766819be2de\") " pod="opendatahub/maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx" Jul 07 18:26:37.283175 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:37.283147 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5mxh\" (UniqueName: \"kubernetes.io/projected/a4e32b23-6587-494c-8e35-0766819be2de-kube-api-access-n5mxh\") pod \"maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx\" (UID: \"a4e32b23-6587-494c-8e35-0766819be2de\") " pod="opendatahub/maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx" Jul 07 18:26:37.396533 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:37.396430 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:37.710268 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:37.710177 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:37.777465 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:37.777425 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4e32b23-6587-494c-8e35-0766819be2de-maas-api-tls\") pod \"maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx\" (UID: \"a4e32b23-6587-494c-8e35-0766819be2de\") " pod="opendatahub/maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx" Jul 07 18:26:37.780508 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:37.780478 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4e32b23-6587-494c-8e35-0766819be2de-maas-api-tls\") pod \"maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx\" (UID: \"a4e32b23-6587-494c-8e35-0766819be2de\") " pod="opendatahub/maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx" Jul 07 18:26:37.814795 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:37.814763 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:37.997733 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:37.997603 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx" Jul 07 18:26:38.166489 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:38.166456 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx"] Jul 07 18:26:38.169269 ip-10-0-136-192 kubenswrapper[2583]: W0707 18:26:38.169239 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4e32b23_6587_494c_8e35_0766819be2de.slice/crio-020fe4ba27609f6a0576527e38ddfa5939d3a3b5825ebb127fd740c60103c2c9 WatchSource:0}: Error finding container 020fe4ba27609f6a0576527e38ddfa5939d3a3b5825ebb127fd740c60103c2c9: Status 404 returned error can't find the container with id 020fe4ba27609f6a0576527e38ddfa5939d3a3b5825ebb127fd740c60103c2c9 Jul 07 18:26:38.872845 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:38.872809 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx" event={"ID":"a4e32b23-6587-494c-8e35-0766819be2de","Type":"ContainerStarted","Data":"fe29b5db858bcadda15baba50006da538b53142529f1dbc9ce7bbbbd9047b2c3"} Jul 07 18:26:38.873062 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:38.872866 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx" event={"ID":"a4e32b23-6587-494c-8e35-0766819be2de","Type":"ContainerStarted","Data":"020fe4ba27609f6a0576527e38ddfa5939d3a3b5825ebb127fd740c60103c2c9"} Jul 07 18:26:38.873062 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:38.872950 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx" Jul 07 18:26:38.894445 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:38.894377 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx" podStartSLOduration=1.5667352220000001 podStartE2EDuration="1.894353231s" totalImagesPullingTime="327.618009ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:26:38.89136073 +0000 UTC m=+1947.973964434" watchObservedRunningTime="2026-07-07 18:26:38.894353231 +0000 UTC m=+1947.976956936" Jul 07 18:26:41.435561 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:41.435527 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:43.026244 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:43.025981 2583 patch_prober.go:28] interesting pod/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": read tcp 10.133.0.2:56762->10.133.0.39:15021: read: connection reset by peer" start-of-body= Jul 07 18:26:43.026244 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:43.026046 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" podUID="42cc52ec-3a4e-4507-b417-7f2013c6c3b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": read tcp 10.133.0.2:56762->10.133.0.39:15021: read: connection reset by peer" Jul 07 18:26:43.923111 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:43.923077 2583 generic.go:356] "Generic (PLEG): container finished" podID="42cc52ec-3a4e-4507-b417-7f2013c6c3b1" containerID="0f982fdff8907ad6dc78b0594de90fb88a8fbf7be274abc18c814cc40264b6e4" exitCode=0 Jul 07 18:26:43.923306 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:43.923152 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" event={"ID":"42cc52ec-3a4e-4507-b417-7f2013c6c3b1","Type":"ContainerDied","Data":"0f982fdff8907ad6dc78b0594de90fb88a8fbf7be274abc18c814cc40264b6e4"} Jul 07 18:26:43.923729 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:43.923712 2583 scope.go:117] "RemoveContainer" containerID="0f982fdff8907ad6dc78b0594de90fb88a8fbf7be274abc18c814cc40264b6e4" Jul 07 18:26:43.925019 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:43.924991 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:26:43.925095 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:43.925079 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:26:43.925143 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:43.925117 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:26:44.885341 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:44.885312 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx" Jul 07 18:26:44.931236 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:44.931136 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" event={"ID":"42cc52ec-3a4e-4507-b417-7f2013c6c3b1","Type":"ContainerStarted","Data":"2270205ded12d383f37bb9fac3ceafd015bf2a4f79c9253ac9908f5386f14551"} Jul 07 18:26:45.753843 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:45.753792 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:45.754113 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:45.754048 2583 patch_prober.go:28] interesting pod/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 07 18:26:45.754171 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:45.754111 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" podUID="42cc52ec-3a4e-4507-b417-7f2013c6c3b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 07 18:26:45.957783 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:45.957742 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:26:46.754014 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:46.753974 2583 patch_prober.go:28] interesting pod/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 07 18:26:46.754217 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:46.754045 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" podUID="42cc52ec-3a4e-4507-b417-7f2013c6c3b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 07 18:26:47.754168 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:47.754064 2583 patch_prober.go:28] interesting pod/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 07 18:26:47.754168 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:47.754139 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" podUID="42cc52ec-3a4e-4507-b417-7f2013c6c3b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 07 18:26:48.754413 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:48.754318 2583 patch_prober.go:28] interesting pod/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 07 18:26:48.754820 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:48.754407 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" podUID="42cc52ec-3a4e-4507-b417-7f2013c6c3b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 07 18:26:48.953977 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:48.953949 2583 generic.go:356] "Generic (PLEG): container finished" podID="d414c637-7f80-4112-bf4c-9e0aefb24133" containerID="21f27cdc9e3ddf06e4ae35d0fd2fa7ac1a27a419997c6b00051bfb29146d8431" exitCode=0 Jul 07 18:26:48.954158 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:48.954025 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" event={"ID":"d414c637-7f80-4112-bf4c-9e0aefb24133","Type":"ContainerDied","Data":"21f27cdc9e3ddf06e4ae35d0fd2fa7ac1a27a419997c6b00051bfb29146d8431"} Jul 07 18:26:48.954480 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:48.954467 2583 scope.go:117] "RemoveContainer" containerID="21f27cdc9e3ddf06e4ae35d0fd2fa7ac1a27a419997c6b00051bfb29146d8431" Jul 07 18:26:48.955523 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:48.955495 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:26:48.955577 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:48.955560 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:26:48.955619 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:48.955599 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:26:49.757253 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:49.757212 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:49.757760 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:49.757642 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:49.758451 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:49.758433 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:26:49.960793 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:49.960761 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" event={"ID":"d414c637-7f80-4112-bf4c-9e0aefb24133","Type":"ContainerStarted","Data":"7795977c587b779a397639a58ae74f91ba9c2f7738f52a2d6b6975d56fd6b355"} Jul 07 18:26:50.322698 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:50.322666 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:50.323756 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:50.323729 2583 patch_prober.go:28] interesting pod/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 07 18:26:50.323892 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:50.323779 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" podUID="d414c637-7f80-4112-bf4c-9e0aefb24133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 07 18:26:50.415768 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:50.415737 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:50.515130 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:50.515097 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:50.855421 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:50.855385 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:51.322927 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:51.322861 2583 patch_prober.go:28] interesting pod/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 07 18:26:51.323088 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:51.322939 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" podUID="d414c637-7f80-4112-bf4c-9e0aefb24133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 07 18:26:52.322663 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:52.322624 2583 patch_prober.go:28] interesting pod/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 07 18:26:52.323131 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:52.322705 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" podUID="d414c637-7f80-4112-bf4c-9e0aefb24133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 07 18:26:53.322165 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:53.322130 2583 patch_prober.go:28] interesting pod/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 07 18:26:53.322343 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:53.322204 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" podUID="d414c637-7f80-4112-bf4c-9e0aefb24133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 07 18:26:54.326915 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:54.326859 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:54.328045 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:54.328020 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:26:54.711629 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:54.711522 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:57.637921 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:57.637865 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:26:58.463893 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:58.463854 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:26:58.564122 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:26:58.564089 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:27:10.247679 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:27:10.247584 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:27:47.513031 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:27:47.512998 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:27:52.850026 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:27:52.849984 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:28:20.779357 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:20.779319 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:28:20.853848 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:20.853818 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:28:21.415438 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:21.415405 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:28:22.405820 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:22.405782 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:28:22.641263 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:22.641230 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:28:23.064677 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:23.064636 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:28:23.233758 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:23.233720 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:28:23.472482 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:23.472387 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:28:23.679648 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:23.679615 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws"] Jul 07 18:28:23.679856 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:23.679831 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws" podUID="86105961-ca05-438b-a080-607318a3b94c" containerName="maas-api" containerID="cri-o://6b297e4dcfc331220eb8acdc37e6ca999482d93474f5372367f7d2f33a31030a" gracePeriod=30 Jul 07 18:28:23.743330 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:23.743242 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:28:23.938306 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:23.938281 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws" Jul 07 18:28:24.058000 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:24.057960 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86105961-ca05-438b-a080-607318a3b94c-openshift-service-ca\") pod \"86105961-ca05-438b-a080-607318a3b94c\" (UID: \"86105961-ca05-438b-a080-607318a3b94c\") " Jul 07 18:28:24.058000 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:24.058003 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6x98\" (UniqueName: \"kubernetes.io/projected/86105961-ca05-438b-a080-607318a3b94c-kube-api-access-r6x98\") pod \"86105961-ca05-438b-a080-607318a3b94c\" (UID: \"86105961-ca05-438b-a080-607318a3b94c\") " Jul 07 18:28:24.058227 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:24.058044 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86105961-ca05-438b-a080-607318a3b94c-maas-api-tls\") pod \"86105961-ca05-438b-a080-607318a3b94c\" (UID: \"86105961-ca05-438b-a080-607318a3b94c\") " Jul 07 18:28:24.058348 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:24.058326 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86105961-ca05-438b-a080-607318a3b94c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "86105961-ca05-438b-a080-607318a3b94c" (UID: "86105961-ca05-438b-a080-607318a3b94c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:28:24.060427 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:24.060404 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86105961-ca05-438b-a080-607318a3b94c-kube-api-access-r6x98" (OuterVolumeSpecName: "kube-api-access-r6x98") pod "86105961-ca05-438b-a080-607318a3b94c" (UID: "86105961-ca05-438b-a080-607318a3b94c"). InnerVolumeSpecName "kube-api-access-r6x98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:28:24.060531 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:24.060404 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86105961-ca05-438b-a080-607318a3b94c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "86105961-ca05-438b-a080-607318a3b94c" (UID: "86105961-ca05-438b-a080-607318a3b94c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:28:24.159366 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:24.159331 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86105961-ca05-438b-a080-607318a3b94c-openshift-service-ca\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:28:24.159366 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:24.159363 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6x98\" (UniqueName: \"kubernetes.io/projected/86105961-ca05-438b-a080-607318a3b94c-kube-api-access-r6x98\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:28:24.159366 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:24.159375 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86105961-ca05-438b-a080-607318a3b94c-maas-api-tls\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:28:24.434207 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:24.434116 2583 generic.go:356] "Generic (PLEG): container finished" podID="86105961-ca05-438b-a080-607318a3b94c" containerID="6b297e4dcfc331220eb8acdc37e6ca999482d93474f5372367f7d2f33a31030a" exitCode=0 Jul 07 18:28:24.434207 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:24.434170 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws" Jul 07 18:28:24.434391 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:24.434203 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws" event={"ID":"86105961-ca05-438b-a080-607318a3b94c","Type":"ContainerDied","Data":"6b297e4dcfc331220eb8acdc37e6ca999482d93474f5372367f7d2f33a31030a"} Jul 07 18:28:24.434391 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:24.434262 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws" event={"ID":"86105961-ca05-438b-a080-607318a3b94c","Type":"ContainerDied","Data":"092dc65e41eca7564a7bd68ebc11f76bd993374a323a7ed36e8e06ac5bdb6ba7"} Jul 07 18:28:24.434391 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:24.434292 2583 scope.go:117] "RemoveContainer" containerID="6b297e4dcfc331220eb8acdc37e6ca999482d93474f5372367f7d2f33a31030a" Jul 07 18:28:24.443694 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:24.443666 2583 scope.go:117] "RemoveContainer" containerID="6b297e4dcfc331220eb8acdc37e6ca999482d93474f5372367f7d2f33a31030a" Jul 07 18:28:24.444051 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:28:24.444028 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b297e4dcfc331220eb8acdc37e6ca999482d93474f5372367f7d2f33a31030a\": container with ID starting with 6b297e4dcfc331220eb8acdc37e6ca999482d93474f5372367f7d2f33a31030a not found: ID does not exist" containerID="6b297e4dcfc331220eb8acdc37e6ca999482d93474f5372367f7d2f33a31030a" Jul 07 18:28:24.444135 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:24.444065 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b297e4dcfc331220eb8acdc37e6ca999482d93474f5372367f7d2f33a31030a"} err="failed to get container status \"6b297e4dcfc331220eb8acdc37e6ca999482d93474f5372367f7d2f33a31030a\": rpc error: code = NotFound desc = could not find container \"6b297e4dcfc331220eb8acdc37e6ca999482d93474f5372367f7d2f33a31030a\": container with ID starting with 6b297e4dcfc331220eb8acdc37e6ca999482d93474f5372367f7d2f33a31030a not found: ID does not exist" Jul 07 18:28:24.462625 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:24.462590 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws"] Jul 07 18:28:24.470451 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:24.470418 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-95ab96-7dc5bdcbc5-7srws"] Jul 07 18:28:25.429118 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:25.429084 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86105961-ca05-438b-a080-607318a3b94c" path="/var/lib/kubelet/pods/86105961-ca05-438b-a080-607318a3b94c/volumes" Jul 07 18:28:25.783108 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:25.783019 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:28:28.113190 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:28.113147 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:28:37.361084 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:37.360998 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:28:51.099468 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:51.099435 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:28:51.204693 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:28:51.204658 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:05.584394 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:05.584359 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:05.842317 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:05.842225 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2"] Jul 07 18:29:05.842495 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:05.842468 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" podUID="42cc52ec-3a4e-4507-b417-7f2013c6c3b1" containerName="istio-proxy" containerID="cri-o://2270205ded12d383f37bb9fac3ceafd015bf2a4f79c9253ac9908f5386f14551" gracePeriod=30 Jul 07 18:29:05.867382 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:05.867352 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:06.108473 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:06.108377 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:06.363915 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:06.363804 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:06.845119 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:06.845076 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:07.012955 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:07.012921 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:07.174474 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:07.174367 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:07.305650 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:07.305616 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx"] Jul 07 18:29:07.305846 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:07.305822 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx" podUID="a4e32b23-6587-494c-8e35-0766819be2de" containerName="maas-api" containerID="cri-o://fe29b5db858bcadda15baba50006da538b53142529f1dbc9ce7bbbbd9047b2c3" gracePeriod=30 Jul 07 18:29:07.344424 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:07.344392 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:07.558793 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:07.558770 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx" Jul 07 18:29:07.590350 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:07.590315 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5mxh\" (UniqueName: \"kubernetes.io/projected/a4e32b23-6587-494c-8e35-0766819be2de-kube-api-access-n5mxh\") pod \"a4e32b23-6587-494c-8e35-0766819be2de\" (UID: \"a4e32b23-6587-494c-8e35-0766819be2de\") " Jul 07 18:29:07.590557 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:07.590425 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4e32b23-6587-494c-8e35-0766819be2de-maas-api-tls\") pod \"a4e32b23-6587-494c-8e35-0766819be2de\" (UID: \"a4e32b23-6587-494c-8e35-0766819be2de\") " Jul 07 18:29:07.590557 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:07.590448 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4e32b23-6587-494c-8e35-0766819be2de-openshift-service-ca\") pod \"a4e32b23-6587-494c-8e35-0766819be2de\" (UID: \"a4e32b23-6587-494c-8e35-0766819be2de\") " Jul 07 18:29:07.590893 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:07.590867 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4e32b23-6587-494c-8e35-0766819be2de-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a4e32b23-6587-494c-8e35-0766819be2de" (UID: "a4e32b23-6587-494c-8e35-0766819be2de"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:29:07.592797 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:07.592767 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4e32b23-6587-494c-8e35-0766819be2de-kube-api-access-n5mxh" (OuterVolumeSpecName: "kube-api-access-n5mxh") pod "a4e32b23-6587-494c-8e35-0766819be2de" (UID: "a4e32b23-6587-494c-8e35-0766819be2de"). InnerVolumeSpecName "kube-api-access-n5mxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:29:07.592970 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:07.592821 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4e32b23-6587-494c-8e35-0766819be2de-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a4e32b23-6587-494c-8e35-0766819be2de" (UID: "a4e32b23-6587-494c-8e35-0766819be2de"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:29:07.658786 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:07.658745 2583 generic.go:356] "Generic (PLEG): container finished" podID="a4e32b23-6587-494c-8e35-0766819be2de" containerID="fe29b5db858bcadda15baba50006da538b53142529f1dbc9ce7bbbbd9047b2c3" exitCode=0 Jul 07 18:29:07.659013 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:07.658796 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx" Jul 07 18:29:07.659013 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:07.658832 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx" event={"ID":"a4e32b23-6587-494c-8e35-0766819be2de","Type":"ContainerDied","Data":"fe29b5db858bcadda15baba50006da538b53142529f1dbc9ce7bbbbd9047b2c3"} Jul 07 18:29:07.659013 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:07.658888 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx" event={"ID":"a4e32b23-6587-494c-8e35-0766819be2de","Type":"ContainerDied","Data":"020fe4ba27609f6a0576527e38ddfa5939d3a3b5825ebb127fd740c60103c2c9"} Jul 07 18:29:07.659013 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:07.658924 2583 scope.go:117] "RemoveContainer" containerID="fe29b5db858bcadda15baba50006da538b53142529f1dbc9ce7bbbbd9047b2c3" Jul 07 18:29:07.668604 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:07.668583 2583 scope.go:117] "RemoveContainer" containerID="fe29b5db858bcadda15baba50006da538b53142529f1dbc9ce7bbbbd9047b2c3" Jul 07 18:29:07.669015 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:29:07.668891 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe29b5db858bcadda15baba50006da538b53142529f1dbc9ce7bbbbd9047b2c3\": container with ID starting with fe29b5db858bcadda15baba50006da538b53142529f1dbc9ce7bbbbd9047b2c3 not found: ID does not exist" containerID="fe29b5db858bcadda15baba50006da538b53142529f1dbc9ce7bbbbd9047b2c3" Jul 07 18:29:07.669094 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:07.669027 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe29b5db858bcadda15baba50006da538b53142529f1dbc9ce7bbbbd9047b2c3"} err="failed to get container status \"fe29b5db858bcadda15baba50006da538b53142529f1dbc9ce7bbbbd9047b2c3\": rpc error: code = NotFound desc = could not find container \"fe29b5db858bcadda15baba50006da538b53142529f1dbc9ce7bbbbd9047b2c3\": container with ID starting with fe29b5db858bcadda15baba50006da538b53142529f1dbc9ce7bbbbd9047b2c3 not found: ID does not exist" Jul 07 18:29:07.685375 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:07.685343 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx"] Jul 07 18:29:07.691442 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:07.691405 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5mxh\" (UniqueName: \"kubernetes.io/projected/a4e32b23-6587-494c-8e35-0766819be2de-kube-api-access-n5mxh\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:07.691442 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:07.691435 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4e32b23-6587-494c-8e35-0766819be2de-maas-api-tls\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:07.691442 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:07.691444 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4e32b23-6587-494c-8e35-0766819be2de-openshift-service-ca\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:07.694368 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:07.694340 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-46d729-784cffc8c8-plcwx"] Jul 07 18:29:09.428763 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:09.428731 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4e32b23-6587-494c-8e35-0766819be2de" path="/var/lib/kubelet/pods/a4e32b23-6587-494c-8e35-0766819be2de/volumes" Jul 07 18:29:09.748789 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:09.748700 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:29:11.103752 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.103686 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:29:11.125934 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.125883 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-credential-socket\") pod \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " Jul 07 18:29:11.126086 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.125959 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-workload-socket\") pod \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " Jul 07 18:29:11.126086 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.125994 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-istio-podinfo\") pod \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " Jul 07 18:29:11.126086 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.126024 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-258t9\" (UniqueName: \"kubernetes.io/projected/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-kube-api-access-258t9\") pod \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " Jul 07 18:29:11.126242 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.126094 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-istio-data\") pod \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " Jul 07 18:29:11.126242 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.126119 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-istio-token\") pod \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " Jul 07 18:29:11.126242 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.126143 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-workload-certs\") pod \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " Jul 07 18:29:11.126242 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.126188 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-istio-envoy\") pod \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " Jul 07 18:29:11.126242 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.126220 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "42cc52ec-3a4e-4507-b417-7f2013c6c3b1" (UID: "42cc52ec-3a4e-4507-b417-7f2013c6c3b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:29:11.126242 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.126231 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-istiod-ca-cert\") pod \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\" (UID: \"42cc52ec-3a4e-4507-b417-7f2013c6c3b1\") " Jul 07 18:29:11.126550 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.126292 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "42cc52ec-3a4e-4507-b417-7f2013c6c3b1" (UID: "42cc52ec-3a4e-4507-b417-7f2013c6c3b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:29:11.126605 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.126574 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-credential-socket\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:11.126605 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.126593 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-workload-socket\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:11.127195 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.127055 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "42cc52ec-3a4e-4507-b417-7f2013c6c3b1" (UID: "42cc52ec-3a4e-4507-b417-7f2013c6c3b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:29:11.127478 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.127435 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-istio-data" (OuterVolumeSpecName: "istio-data") pod "42cc52ec-3a4e-4507-b417-7f2013c6c3b1" (UID: "42cc52ec-3a4e-4507-b417-7f2013c6c3b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:29:11.129650 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.128311 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "42cc52ec-3a4e-4507-b417-7f2013c6c3b1" (UID: "42cc52ec-3a4e-4507-b417-7f2013c6c3b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:29:11.129782 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.129742 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "42cc52ec-3a4e-4507-b417-7f2013c6c3b1" (UID: "42cc52ec-3a4e-4507-b417-7f2013c6c3b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:29:11.129960 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.129847 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-istio-token" (OuterVolumeSpecName: "istio-token") pod "42cc52ec-3a4e-4507-b417-7f2013c6c3b1" (UID: "42cc52ec-3a4e-4507-b417-7f2013c6c3b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:29:11.130037 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.129983 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "42cc52ec-3a4e-4507-b417-7f2013c6c3b1" (UID: "42cc52ec-3a4e-4507-b417-7f2013c6c3b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 18:29:11.132021 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.131995 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-kube-api-access-258t9" (OuterVolumeSpecName: "kube-api-access-258t9") pod "42cc52ec-3a4e-4507-b417-7f2013c6c3b1" (UID: "42cc52ec-3a4e-4507-b417-7f2013c6c3b1"). InnerVolumeSpecName "kube-api-access-258t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:29:11.228033 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.227985 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-istio-data\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:11.228033 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.228025 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-istio-token\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:11.228033 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.228037 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-workload-certs\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:11.228294 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.228048 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-istio-envoy\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:11.228294 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.228061 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-istiod-ca-cert\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:11.228294 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.228072 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-istio-podinfo\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:11.228294 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.228083 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-258t9\" (UniqueName: \"kubernetes.io/projected/42cc52ec-3a4e-4507-b417-7f2013c6c3b1-kube-api-access-258t9\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:11.492983 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.492954 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wqfm4_997df85c-7632-47b5-a8a6-1c37f3346251/console-operator/2.log" Jul 07 18:29:11.493941 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.493914 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wqfm4_997df85c-7632-47b5-a8a6-1c37f3346251/console-operator/2.log" Jul 07 18:29:11.500015 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.499989 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zjzpw_5c2d819c-e495-47ec-a1e7-4b2582a195bd/ovn-acl-logging/0.log" Jul 07 18:29:11.501196 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.501171 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zjzpw_5c2d819c-e495-47ec-a1e7-4b2582a195bd/ovn-acl-logging/0.log" Jul 07 18:29:11.682504 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.682471 2583 generic.go:356] "Generic (PLEG): container finished" podID="42cc52ec-3a4e-4507-b417-7f2013c6c3b1" containerID="2270205ded12d383f37bb9fac3ceafd015bf2a4f79c9253ac9908f5386f14551" exitCode=0 Jul 07 18:29:11.682836 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.682535 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" Jul 07 18:29:11.682836 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.682549 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" event={"ID":"42cc52ec-3a4e-4507-b417-7f2013c6c3b1","Type":"ContainerDied","Data":"2270205ded12d383f37bb9fac3ceafd015bf2a4f79c9253ac9908f5386f14551"} Jul 07 18:29:11.682836 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.682604 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2" event={"ID":"42cc52ec-3a4e-4507-b417-7f2013c6c3b1","Type":"ContainerDied","Data":"aa318875cebf3b2e563683dcf5e827160cd664a91a9999a79e697b6cf06d10b2"} Jul 07 18:29:11.682836 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.682627 2583 scope.go:117] "RemoveContainer" containerID="2270205ded12d383f37bb9fac3ceafd015bf2a4f79c9253ac9908f5386f14551" Jul 07 18:29:11.691589 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.691557 2583 scope.go:117] "RemoveContainer" containerID="0f982fdff8907ad6dc78b0594de90fb88a8fbf7be274abc18c814cc40264b6e4" Jul 07 18:29:11.700345 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.700325 2583 scope.go:117] "RemoveContainer" containerID="2270205ded12d383f37bb9fac3ceafd015bf2a4f79c9253ac9908f5386f14551" Jul 07 18:29:11.700654 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:29:11.700634 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2270205ded12d383f37bb9fac3ceafd015bf2a4f79c9253ac9908f5386f14551\": container with ID starting with 2270205ded12d383f37bb9fac3ceafd015bf2a4f79c9253ac9908f5386f14551 not found: ID does not exist" containerID="2270205ded12d383f37bb9fac3ceafd015bf2a4f79c9253ac9908f5386f14551" Jul 07 18:29:11.700700 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.700666 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2270205ded12d383f37bb9fac3ceafd015bf2a4f79c9253ac9908f5386f14551"} err="failed to get container status \"2270205ded12d383f37bb9fac3ceafd015bf2a4f79c9253ac9908f5386f14551\": rpc error: code = NotFound desc = could not find container \"2270205ded12d383f37bb9fac3ceafd015bf2a4f79c9253ac9908f5386f14551\": container with ID starting with 2270205ded12d383f37bb9fac3ceafd015bf2a4f79c9253ac9908f5386f14551 not found: ID does not exist" Jul 07 18:29:11.700700 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.700685 2583 scope.go:117] "RemoveContainer" containerID="0f982fdff8907ad6dc78b0594de90fb88a8fbf7be274abc18c814cc40264b6e4" Jul 07 18:29:11.701020 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:29:11.701000 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f982fdff8907ad6dc78b0594de90fb88a8fbf7be274abc18c814cc40264b6e4\": container with ID starting with 0f982fdff8907ad6dc78b0594de90fb88a8fbf7be274abc18c814cc40264b6e4 not found: ID does not exist" containerID="0f982fdff8907ad6dc78b0594de90fb88a8fbf7be274abc18c814cc40264b6e4" Jul 07 18:29:11.701083 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.701025 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f982fdff8907ad6dc78b0594de90fb88a8fbf7be274abc18c814cc40264b6e4"} err="failed to get container status \"0f982fdff8907ad6dc78b0594de90fb88a8fbf7be274abc18c814cc40264b6e4\": rpc error: code = NotFound desc = could not find container \"0f982fdff8907ad6dc78b0594de90fb88a8fbf7be274abc18c814cc40264b6e4\": container with ID starting with 0f982fdff8907ad6dc78b0594de90fb88a8fbf7be274abc18c814cc40264b6e4 not found: ID does not exist" Jul 07 18:29:11.704783 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.704755 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2"] Jul 07 18:29:11.709264 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:11.709235 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-95ab96-openshift-default-cd8d5c5c-tqgp2"] Jul 07 18:29:13.429518 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:13.429482 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42cc52ec-3a4e-4507-b417-7f2013c6c3b1" path="/var/lib/kubelet/pods/42cc52ec-3a4e-4507-b417-7f2013c6c3b1/volumes" Jul 07 18:29:13.754913 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:13.754811 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:13.854046 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:13.854009 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:14.024968 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:14.024853 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8"] Jul 07 18:29:14.025137 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:14.025107 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" podUID="d414c637-7f80-4112-bf4c-9e0aefb24133" containerName="istio-proxy" containerID="cri-o://7795977c587b779a397639a58ae74f91ba9c2f7738f52a2d6b6975d56fd6b355" gracePeriod=30 Jul 07 18:29:14.051091 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:14.051053 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:14.291836 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:14.291749 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:14.647996 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:14.647964 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:14.957221 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:14.957122 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6"] Jul 07 18:29:14.957618 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:14.957596 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42cc52ec-3a4e-4507-b417-7f2013c6c3b1" containerName="istio-proxy" Jul 07 18:29:14.957618 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:14.957617 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="42cc52ec-3a4e-4507-b417-7f2013c6c3b1" containerName="istio-proxy" Jul 07 18:29:14.957792 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:14.957628 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4e32b23-6587-494c-8e35-0766819be2de" containerName="maas-api" Jul 07 18:29:14.957792 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:14.957633 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4e32b23-6587-494c-8e35-0766819be2de" containerName="maas-api" Jul 07 18:29:14.957792 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:14.957648 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86105961-ca05-438b-a080-607318a3b94c" containerName="maas-api" Jul 07 18:29:14.957792 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:14.957654 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="86105961-ca05-438b-a080-607318a3b94c" containerName="maas-api" Jul 07 18:29:14.957792 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:14.957726 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="42cc52ec-3a4e-4507-b417-7f2013c6c3b1" containerName="istio-proxy" Jul 07 18:29:14.957792 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:14.957734 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="86105961-ca05-438b-a080-607318a3b94c" containerName="maas-api" Jul 07 18:29:14.957792 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:14.957748 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4e32b23-6587-494c-8e35-0766819be2de" containerName="maas-api" Jul 07 18:29:14.960756 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:14.960739 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:14.963331 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:14.963311 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-3e6980-openshift-default-dockercfg-k82rn\"" Jul 07 18:29:14.973174 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:14.973148 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6"] Jul 07 18:29:14.984027 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:14.983891 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:15.064939 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.064883 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-istio-data\") pod \"e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:15.065419 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.065390 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-workload-socket\") pod \"e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:15.065590 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.065569 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-istio-token\") pod \"e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:15.065720 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.065608 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-workload-certs\") pod \"e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:15.065720 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.065679 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpf95\" (UniqueName: \"kubernetes.io/projected/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-kube-api-access-mpf95\") pod \"e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:15.065843 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.065729 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-credential-socket\") pod \"e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:15.065843 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.065803 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-istio-envoy\") pod \"e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:15.065843 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.065831 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-istiod-ca-cert\") pod \"e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:15.065998 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.065925 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-istio-podinfo\") pod \"e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:15.140441 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.140408 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:15.166733 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.166694 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-istio-data\") pod \"e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:15.166733 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.166735 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-workload-socket\") pod \"e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:15.167010 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.166757 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-istio-token\") pod \"e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:15.167010 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.166785 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-workload-certs\") pod \"e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:15.167010 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.166822 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpf95\" (UniqueName: \"kubernetes.io/projected/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-kube-api-access-mpf95\") pod \"e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:15.167010 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.166854 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-credential-socket\") pod \"e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:15.167010 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.166958 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-istio-envoy\") pod \"e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:15.167246 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.167037 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-istiod-ca-cert\") pod \"e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:15.167246 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.167107 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-istio-podinfo\") pod \"e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:15.167246 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.167184 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-istio-data\") pod \"e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:15.167246 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.167229 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-workload-socket\") pod \"e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:15.167458 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.167361 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-workload-certs\") pod \"e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:15.167458 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.167430 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-credential-socket\") pod \"e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:15.167726 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.167707 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-istiod-ca-cert\") pod \"e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:15.169501 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.169483 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-istio-envoy\") pod \"e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:15.169841 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.169821 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-istio-podinfo\") pod \"e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:15.175797 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.175766 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-istio-token\") pod \"e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:15.175933 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.175865 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpf95\" (UniqueName: \"kubernetes.io/projected/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-kube-api-access-mpf95\") pod \"e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:15.271142 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.271042 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:15.415972 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.415946 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6"] Jul 07 18:29:15.418626 ip-10-0-136-192 kubenswrapper[2583]: W0707 18:29:15.418590 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd459071e_6ede_4bf7_ac3b_b5c0fb0a0d17.slice/crio-b689eac2be4cc53fd7481ee50f3c4f77ad3a5a59d46f3641924d599fb3a20d9d WatchSource:0}: Error finding container b689eac2be4cc53fd7481ee50f3c4f77ad3a5a59d46f3641924d599fb3a20d9d: Status 404 returned error can't find the container with id b689eac2be4cc53fd7481ee50f3c4f77ad3a5a59d46f3641924d599fb3a20d9d Jul 07 18:29:15.422939 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.421391 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:29:15.422939 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.421562 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:29:15.422939 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.421653 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:29:15.705869 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.705830 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" event={"ID":"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17","Type":"ContainerStarted","Data":"685b2fa49ec57e7fb598e069765198b7f0b0f3f7f79b825b6594747a906def62"} Jul 07 18:29:15.706377 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.705887 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" event={"ID":"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17","Type":"ContainerStarted","Data":"b689eac2be4cc53fd7481ee50f3c4f77ad3a5a59d46f3641924d599fb3a20d9d"} Jul 07 18:29:15.731276 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:15.731200 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" podStartSLOduration=1.731178192 podStartE2EDuration="1.731178192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:29:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:29:15.731157394 +0000 UTC m=+2104.813761099" watchObservedRunningTime="2026-07-07 18:29:15.731178192 +0000 UTC m=+2104.813781909" Jul 07 18:29:16.271722 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:16.271683 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:16.276681 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:16.276655 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:16.711204 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:16.711174 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:16.712318 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:16.712297 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:17.802553 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:17.802518 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:17.959137 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:17.959104 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt"] Jul 07 18:29:17.959513 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:17.959500 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42cc52ec-3a4e-4507-b417-7f2013c6c3b1" containerName="istio-proxy" Jul 07 18:29:17.959558 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:17.959515 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="42cc52ec-3a4e-4507-b417-7f2013c6c3b1" containerName="istio-proxy" Jul 07 18:29:17.959595 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:17.959581 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="42cc52ec-3a4e-4507-b417-7f2013c6c3b1" containerName="istio-proxy" Jul 07 18:29:17.962663 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:17.962647 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt" Jul 07 18:29:17.966253 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:17.966231 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-jnv2d\"" Jul 07 18:29:17.966489 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:17.966473 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-3e6980-serving-cert\"" Jul 07 18:29:17.985255 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:17.985226 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt"] Jul 07 18:29:18.013485 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:18.013454 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:18.096291 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:18.096193 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2q79\" (UniqueName: \"kubernetes.io/projected/44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0-kube-api-access-t2q79\") pod \"maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt\" (UID: \"44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0\") " pod="opendatahub/maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt" Jul 07 18:29:18.096291 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:18.096236 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0-maas-api-tls\") pod \"maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt\" (UID: \"44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0\") " pod="opendatahub/maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt" Jul 07 18:29:18.096487 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:18.096363 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0-openshift-service-ca\") pod \"maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt\" (UID: \"44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0\") " pod="opendatahub/maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt" Jul 07 18:29:18.113522 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:18.113485 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:18.197577 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:18.197532 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0-openshift-service-ca\") pod \"maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt\" (UID: \"44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0\") " pod="opendatahub/maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt" Jul 07 18:29:18.197761 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:18.197619 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2q79\" (UniqueName: \"kubernetes.io/projected/44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0-kube-api-access-t2q79\") pod \"maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt\" (UID: \"44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0\") " pod="opendatahub/maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt" Jul 07 18:29:18.197761 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:18.197647 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0-maas-api-tls\") pod \"maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt\" (UID: \"44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0\") " pod="opendatahub/maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt" Jul 07 18:29:18.198317 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:18.198292 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0-openshift-service-ca\") pod \"maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt\" (UID: \"44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0\") " pod="opendatahub/maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt" Jul 07 18:29:18.200301 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:18.200278 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0-maas-api-tls\") pod \"maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt\" (UID: \"44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0\") " pod="opendatahub/maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt" Jul 07 18:29:18.207477 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:18.207448 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2q79\" (UniqueName: \"kubernetes.io/projected/44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0-kube-api-access-t2q79\") pod \"maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt\" (UID: \"44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0\") " pod="opendatahub/maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt" Jul 07 18:29:18.273093 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:18.273052 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt" Jul 07 18:29:18.413058 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:18.413033 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt"] Jul 07 18:29:18.415105 ip-10-0-136-192 kubenswrapper[2583]: W0707 18:29:18.415073 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d0c9d1_8ae7_4f4d_a971_474bc6f22dd0.slice/crio-8e59e6ef70fe11005003f88a389289d18d59a4680fbc68723eb994370386fbc0 WatchSource:0}: Error finding container 8e59e6ef70fe11005003f88a389289d18d59a4680fbc68723eb994370386fbc0: Status 404 returned error can't find the container with id 8e59e6ef70fe11005003f88a389289d18d59a4680fbc68723eb994370386fbc0 Jul 07 18:29:18.599123 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:18.599090 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:18.700120 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:18.700022 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:18.721836 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:18.721803 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt" event={"ID":"44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0","Type":"ContainerStarted","Data":"8e59e6ef70fe11005003f88a389289d18d59a4680fbc68723eb994370386fbc0"} Jul 07 18:29:19.278967 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.278943 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:29:19.411525 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.411483 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d414c637-7f80-4112-bf4c-9e0aefb24133-workload-socket\") pod \"d414c637-7f80-4112-bf4c-9e0aefb24133\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " Jul 07 18:29:19.411525 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.411525 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d414c637-7f80-4112-bf4c-9e0aefb24133-istiod-ca-cert\") pod \"d414c637-7f80-4112-bf4c-9e0aefb24133\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " Jul 07 18:29:19.411787 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.411588 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d414c637-7f80-4112-bf4c-9e0aefb24133-istio-envoy\") pod \"d414c637-7f80-4112-bf4c-9e0aefb24133\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " Jul 07 18:29:19.411787 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.411617 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d414c637-7f80-4112-bf4c-9e0aefb24133-istio-podinfo\") pod \"d414c637-7f80-4112-bf4c-9e0aefb24133\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " Jul 07 18:29:19.411787 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.411640 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mj47m\" (UniqueName: \"kubernetes.io/projected/d414c637-7f80-4112-bf4c-9e0aefb24133-kube-api-access-mj47m\") pod \"d414c637-7f80-4112-bf4c-9e0aefb24133\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " Jul 07 18:29:19.411787 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.411674 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d414c637-7f80-4112-bf4c-9e0aefb24133-istio-token\") pod \"d414c637-7f80-4112-bf4c-9e0aefb24133\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " Jul 07 18:29:19.411787 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.411701 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d414c637-7f80-4112-bf4c-9e0aefb24133-credential-socket\") pod \"d414c637-7f80-4112-bf4c-9e0aefb24133\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " Jul 07 18:29:19.411787 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.411740 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d414c637-7f80-4112-bf4c-9e0aefb24133-workload-certs\") pod \"d414c637-7f80-4112-bf4c-9e0aefb24133\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " Jul 07 18:29:19.411787 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.411770 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d414c637-7f80-4112-bf4c-9e0aefb24133-istio-data\") pod \"d414c637-7f80-4112-bf4c-9e0aefb24133\" (UID: \"d414c637-7f80-4112-bf4c-9e0aefb24133\") " Jul 07 18:29:19.412176 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.411849 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d414c637-7f80-4112-bf4c-9e0aefb24133-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d414c637-7f80-4112-bf4c-9e0aefb24133" (UID: "d414c637-7f80-4112-bf4c-9e0aefb24133"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:29:19.412176 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.412094 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d414c637-7f80-4112-bf4c-9e0aefb24133-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d414c637-7f80-4112-bf4c-9e0aefb24133" (UID: "d414c637-7f80-4112-bf4c-9e0aefb24133"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:29:19.412176 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.412120 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d414c637-7f80-4112-bf4c-9e0aefb24133-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d414c637-7f80-4112-bf4c-9e0aefb24133" (UID: "d414c637-7f80-4112-bf4c-9e0aefb24133"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:29:19.412400 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.412193 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d414c637-7f80-4112-bf4c-9e0aefb24133-workload-socket\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:19.412400 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.412233 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d414c637-7f80-4112-bf4c-9e0aefb24133-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d414c637-7f80-4112-bf4c-9e0aefb24133" (UID: "d414c637-7f80-4112-bf4c-9e0aefb24133"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:29:19.413133 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.413105 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d414c637-7f80-4112-bf4c-9e0aefb24133-istio-data" (OuterVolumeSpecName: "istio-data") pod "d414c637-7f80-4112-bf4c-9e0aefb24133" (UID: "d414c637-7f80-4112-bf4c-9e0aefb24133"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:29:19.414141 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.414114 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d414c637-7f80-4112-bf4c-9e0aefb24133-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d414c637-7f80-4112-bf4c-9e0aefb24133" (UID: "d414c637-7f80-4112-bf4c-9e0aefb24133"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:29:19.414248 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.414200 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d414c637-7f80-4112-bf4c-9e0aefb24133-istio-token" (OuterVolumeSpecName: "istio-token") pod "d414c637-7f80-4112-bf4c-9e0aefb24133" (UID: "d414c637-7f80-4112-bf4c-9e0aefb24133"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:29:19.414301 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.414253 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d414c637-7f80-4112-bf4c-9e0aefb24133-kube-api-access-mj47m" (OuterVolumeSpecName: "kube-api-access-mj47m") pod "d414c637-7f80-4112-bf4c-9e0aefb24133" (UID: "d414c637-7f80-4112-bf4c-9e0aefb24133"). InnerVolumeSpecName "kube-api-access-mj47m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:29:19.414419 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.414403 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d414c637-7f80-4112-bf4c-9e0aefb24133-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d414c637-7f80-4112-bf4c-9e0aefb24133" (UID: "d414c637-7f80-4112-bf4c-9e0aefb24133"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 18:29:19.512962 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.512925 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d414c637-7f80-4112-bf4c-9e0aefb24133-istio-envoy\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:19.512962 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.512957 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d414c637-7f80-4112-bf4c-9e0aefb24133-istio-podinfo\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:19.513166 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.512990 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj47m\" (UniqueName: \"kubernetes.io/projected/d414c637-7f80-4112-bf4c-9e0aefb24133-kube-api-access-mj47m\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:19.513166 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.513000 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d414c637-7f80-4112-bf4c-9e0aefb24133-istio-token\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:19.513166 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.513009 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d414c637-7f80-4112-bf4c-9e0aefb24133-credential-socket\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:19.513166 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.513017 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d414c637-7f80-4112-bf4c-9e0aefb24133-workload-certs\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:19.513166 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.513026 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d414c637-7f80-4112-bf4c-9e0aefb24133-istio-data\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:19.513166 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.513034 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d414c637-7f80-4112-bf4c-9e0aefb24133-istiod-ca-cert\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:19.728479 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.728383 2583 generic.go:356] "Generic (PLEG): container finished" podID="d414c637-7f80-4112-bf4c-9e0aefb24133" containerID="7795977c587b779a397639a58ae74f91ba9c2f7738f52a2d6b6975d56fd6b355" exitCode=0 Jul 07 18:29:19.728479 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.728453 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" Jul 07 18:29:19.728719 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.728471 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" event={"ID":"d414c637-7f80-4112-bf4c-9e0aefb24133","Type":"ContainerDied","Data":"7795977c587b779a397639a58ae74f91ba9c2f7738f52a2d6b6975d56fd6b355"} Jul 07 18:29:19.728719 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.728530 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8" event={"ID":"d414c637-7f80-4112-bf4c-9e0aefb24133","Type":"ContainerDied","Data":"ace1ffec2f0d01454dc664b40321545979f1d393d48d56245b64fb8eb4903cd3"} Jul 07 18:29:19.728719 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.728559 2583 scope.go:117] "RemoveContainer" containerID="7795977c587b779a397639a58ae74f91ba9c2f7738f52a2d6b6975d56fd6b355" Jul 07 18:29:19.730639 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.730610 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt" event={"ID":"44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0","Type":"ContainerStarted","Data":"dacc2ce971e29824e9212fc788eb70e90c0c5d9a02dc1c9732d3f3ee3083bf9c"} Jul 07 18:29:19.731260 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.731208 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt" Jul 07 18:29:19.737671 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.737650 2583 scope.go:117] "RemoveContainer" containerID="21f27cdc9e3ddf06e4ae35d0fd2fa7ac1a27a419997c6b00051bfb29146d8431" Jul 07 18:29:19.751314 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.751284 2583 scope.go:117] "RemoveContainer" containerID="7795977c587b779a397639a58ae74f91ba9c2f7738f52a2d6b6975d56fd6b355" Jul 07 18:29:19.751559 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.751530 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8"] Jul 07 18:29:19.751703 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:29:19.751673 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7795977c587b779a397639a58ae74f91ba9c2f7738f52a2d6b6975d56fd6b355\": container with ID starting with 7795977c587b779a397639a58ae74f91ba9c2f7738f52a2d6b6975d56fd6b355 not found: ID does not exist" containerID="7795977c587b779a397639a58ae74f91ba9c2f7738f52a2d6b6975d56fd6b355" Jul 07 18:29:19.751759 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.751720 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7795977c587b779a397639a58ae74f91ba9c2f7738f52a2d6b6975d56fd6b355"} err="failed to get container status \"7795977c587b779a397639a58ae74f91ba9c2f7738f52a2d6b6975d56fd6b355\": rpc error: code = NotFound desc = could not find container \"7795977c587b779a397639a58ae74f91ba9c2f7738f52a2d6b6975d56fd6b355\": container with ID starting with 7795977c587b779a397639a58ae74f91ba9c2f7738f52a2d6b6975d56fd6b355 not found: ID does not exist" Jul 07 18:29:19.751804 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.751764 2583 scope.go:117] "RemoveContainer" containerID="21f27cdc9e3ddf06e4ae35d0fd2fa7ac1a27a419997c6b00051bfb29146d8431" Jul 07 18:29:19.752114 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:29:19.752082 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21f27cdc9e3ddf06e4ae35d0fd2fa7ac1a27a419997c6b00051bfb29146d8431\": container with ID starting with 21f27cdc9e3ddf06e4ae35d0fd2fa7ac1a27a419997c6b00051bfb29146d8431 not found: ID does not exist" containerID="21f27cdc9e3ddf06e4ae35d0fd2fa7ac1a27a419997c6b00051bfb29146d8431" Jul 07 18:29:19.752167 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.752120 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21f27cdc9e3ddf06e4ae35d0fd2fa7ac1a27a419997c6b00051bfb29146d8431"} err="failed to get container status \"21f27cdc9e3ddf06e4ae35d0fd2fa7ac1a27a419997c6b00051bfb29146d8431\": rpc error: code = NotFound desc = could not find container \"21f27cdc9e3ddf06e4ae35d0fd2fa7ac1a27a419997c6b00051bfb29146d8431\": container with ID starting with 21f27cdc9e3ddf06e4ae35d0fd2fa7ac1a27a419997c6b00051bfb29146d8431 not found: ID does not exist" Jul 07 18:29:19.753430 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.753412 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-46d729-openshift-default-84f89866bc-fq2f8"] Jul 07 18:29:19.767683 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:19.767634 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt" podStartSLOduration=2.357338965 podStartE2EDuration="2.767617185s" totalImagesPullingTime="410.27822ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:29:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:29:19.766357844 +0000 UTC m=+2108.848961542" watchObservedRunningTime="2026-07-07 18:29:19.767617185 +0000 UTC m=+2108.850220905" Jul 07 18:29:21.430831 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:21.430791 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d414c637-7f80-4112-bf4c-9e0aefb24133" path="/var/lib/kubelet/pods/d414c637-7f80-4112-bf4c-9e0aefb24133/volumes" Jul 07 18:29:22.255823 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:22.255785 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:29:22.365126 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:22.365093 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:22.868658 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:22.868624 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:23.220136 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.220032 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx"] Jul 07 18:29:23.220550 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.220530 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d414c637-7f80-4112-bf4c-9e0aefb24133" containerName="istio-proxy" Jul 07 18:29:23.220648 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.220554 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d414c637-7f80-4112-bf4c-9e0aefb24133" containerName="istio-proxy" Jul 07 18:29:23.220648 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.220575 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d414c637-7f80-4112-bf4c-9e0aefb24133" containerName="istio-proxy" Jul 07 18:29:23.220648 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.220582 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d414c637-7f80-4112-bf4c-9e0aefb24133" containerName="istio-proxy" Jul 07 18:29:23.220799 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.220685 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d414c637-7f80-4112-bf4c-9e0aefb24133" containerName="istio-proxy" Jul 07 18:29:23.220799 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.220701 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d414c637-7f80-4112-bf4c-9e0aefb24133" containerName="istio-proxy" Jul 07 18:29:23.225508 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.225483 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:23.236242 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.236000 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-06e51f-openshift-default-dockercfg-sshd4\"" Jul 07 18:29:23.238027 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.238003 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx"] Jul 07 18:29:23.269475 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.269435 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:23.349966 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.349925 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4610ded5-b188-438f-9067-0ee96bd33107-credential-socket\") pod \"e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:23.350139 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.350015 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79r7m\" (UniqueName: \"kubernetes.io/projected/4610ded5-b188-438f-9067-0ee96bd33107-kube-api-access-79r7m\") pod \"e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:23.350139 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.350042 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4610ded5-b188-438f-9067-0ee96bd33107-istio-data\") pod \"e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:23.350139 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.350076 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4610ded5-b188-438f-9067-0ee96bd33107-istio-envoy\") pod \"e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:23.350139 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.350125 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4610ded5-b188-438f-9067-0ee96bd33107-workload-socket\") pod \"e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:23.350336 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.350154 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4610ded5-b188-438f-9067-0ee96bd33107-istio-token\") pod \"e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:23.350336 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.350177 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4610ded5-b188-438f-9067-0ee96bd33107-workload-certs\") pod \"e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:23.350336 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.350256 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4610ded5-b188-438f-9067-0ee96bd33107-istio-podinfo\") pod \"e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:23.350336 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.350320 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4610ded5-b188-438f-9067-0ee96bd33107-istiod-ca-cert\") pod \"e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:23.451758 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.451719 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79r7m\" (UniqueName: \"kubernetes.io/projected/4610ded5-b188-438f-9067-0ee96bd33107-kube-api-access-79r7m\") pod \"e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:23.451758 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.451758 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4610ded5-b188-438f-9067-0ee96bd33107-istio-data\") pod \"e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:23.452054 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.451789 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4610ded5-b188-438f-9067-0ee96bd33107-istio-envoy\") pod \"e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:23.452054 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.451854 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4610ded5-b188-438f-9067-0ee96bd33107-workload-socket\") pod \"e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:23.452054 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.451882 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4610ded5-b188-438f-9067-0ee96bd33107-istio-token\") pod \"e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:23.452054 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.451934 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4610ded5-b188-438f-9067-0ee96bd33107-workload-certs\") pod \"e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:23.452054 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.451967 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4610ded5-b188-438f-9067-0ee96bd33107-istio-podinfo\") pod \"e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:23.452054 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.452015 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4610ded5-b188-438f-9067-0ee96bd33107-istiod-ca-cert\") pod \"e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:23.452359 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.452061 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4610ded5-b188-438f-9067-0ee96bd33107-credential-socket\") pod \"e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:23.452359 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.452266 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4610ded5-b188-438f-9067-0ee96bd33107-istio-data\") pod \"e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:23.452359 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.452299 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4610ded5-b188-438f-9067-0ee96bd33107-workload-socket\") pod \"e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:23.452567 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.452549 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4610ded5-b188-438f-9067-0ee96bd33107-workload-certs\") pod \"e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:23.452644 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.452549 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4610ded5-b188-438f-9067-0ee96bd33107-credential-socket\") pod \"e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:23.452791 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.452770 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4610ded5-b188-438f-9067-0ee96bd33107-istiod-ca-cert\") pod \"e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:23.454311 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.454293 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4610ded5-b188-438f-9067-0ee96bd33107-istio-envoy\") pod \"e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:23.454571 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.454553 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4610ded5-b188-438f-9067-0ee96bd33107-istio-podinfo\") pod \"e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:23.460535 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.460508 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4610ded5-b188-438f-9067-0ee96bd33107-istio-token\") pod \"e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:23.460648 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.460630 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79r7m\" (UniqueName: \"kubernetes.io/projected/4610ded5-b188-438f-9067-0ee96bd33107-kube-api-access-79r7m\") pod \"e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:23.537977 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.537868 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:23.714732 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.714617 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:23.737284 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.737211 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx"] Jul 07 18:29:23.739599 ip-10-0-136-192 kubenswrapper[2583]: W0707 18:29:23.739572 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4610ded5_b188_438f_9067_0ee96bd33107.slice/crio-b86b45ccbf4f2201713df5ecaa7b999e9b072a58416c8494b871f6d4613e4e90 WatchSource:0}: Error finding container b86b45ccbf4f2201713df5ecaa7b999e9b072a58416c8494b871f6d4613e4e90: Status 404 returned error can't find the container with id b86b45ccbf4f2201713df5ecaa7b999e9b072a58416c8494b871f6d4613e4e90 Jul 07 18:29:23.741879 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.741837 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:29:23.741986 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.741951 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:29:23.742041 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.742001 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:29:23.754328 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:23.754300 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" event={"ID":"4610ded5-b188-438f-9067-0ee96bd33107","Type":"ContainerStarted","Data":"b86b45ccbf4f2201713df5ecaa7b999e9b072a58416c8494b871f6d4613e4e90"} Jul 07 18:29:24.765144 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:24.762769 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" event={"ID":"4610ded5-b188-438f-9067-0ee96bd33107","Type":"ContainerStarted","Data":"f474684cd23da2a32898da4edd828761124422856c0675f9355414197d43d52c"} Jul 07 18:29:24.821862 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:24.821805 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" podStartSLOduration=1.8217892770000002 podStartE2EDuration="1.821789277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:29:24.820604164 +0000 UTC m=+2113.903207868" watchObservedRunningTime="2026-07-07 18:29:24.821789277 +0000 UTC m=+2113.904393044" Jul 07 18:29:25.538349 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:25.538308 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:25.543320 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:25.543293 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:25.767630 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:25.767592 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:25.768796 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:25.768775 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:26.748661 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:26.748631 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt" Jul 07 18:29:26.877499 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:26.877455 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:26.954967 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:26.954931 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-06e51f-6979b974-w92xw"] Jul 07 18:29:26.958391 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:26.958367 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-06e51f-6979b974-w92xw" Jul 07 18:29:26.962192 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:26.962168 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-06e51f-serving-cert\"" Jul 07 18:29:26.975061 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:26.975033 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-06e51f-6979b974-w92xw"] Jul 07 18:29:26.981758 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:26.981730 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:27.076739 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:27.076701 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:27.097927 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:27.096479 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn2lx\" (UniqueName: \"kubernetes.io/projected/8a71b337-5f22-43c3-899a-74dbdfad9057-kube-api-access-hn2lx\") pod \"maas-api-e2e-api-b-06e51f-6979b974-w92xw\" (UID: \"8a71b337-5f22-43c3-899a-74dbdfad9057\") " pod="opendatahub/maas-api-e2e-api-b-06e51f-6979b974-w92xw" Jul 07 18:29:27.097927 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:27.096549 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a71b337-5f22-43c3-899a-74dbdfad9057-maas-api-tls\") pod \"maas-api-e2e-api-b-06e51f-6979b974-w92xw\" (UID: \"8a71b337-5f22-43c3-899a-74dbdfad9057\") " pod="opendatahub/maas-api-e2e-api-b-06e51f-6979b974-w92xw" Jul 07 18:29:27.097927 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:27.096583 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a71b337-5f22-43c3-899a-74dbdfad9057-openshift-service-ca\") pod \"maas-api-e2e-api-b-06e51f-6979b974-w92xw\" (UID: \"8a71b337-5f22-43c3-899a-74dbdfad9057\") " pod="opendatahub/maas-api-e2e-api-b-06e51f-6979b974-w92xw" Jul 07 18:29:27.197633 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:27.197559 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn2lx\" (UniqueName: \"kubernetes.io/projected/8a71b337-5f22-43c3-899a-74dbdfad9057-kube-api-access-hn2lx\") pod \"maas-api-e2e-api-b-06e51f-6979b974-w92xw\" (UID: \"8a71b337-5f22-43c3-899a-74dbdfad9057\") " pod="opendatahub/maas-api-e2e-api-b-06e51f-6979b974-w92xw" Jul 07 18:29:27.197821 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:27.197656 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a71b337-5f22-43c3-899a-74dbdfad9057-maas-api-tls\") pod \"maas-api-e2e-api-b-06e51f-6979b974-w92xw\" (UID: \"8a71b337-5f22-43c3-899a-74dbdfad9057\") " pod="opendatahub/maas-api-e2e-api-b-06e51f-6979b974-w92xw" Jul 07 18:29:27.197821 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:27.197688 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a71b337-5f22-43c3-899a-74dbdfad9057-openshift-service-ca\") pod \"maas-api-e2e-api-b-06e51f-6979b974-w92xw\" (UID: \"8a71b337-5f22-43c3-899a-74dbdfad9057\") " pod="opendatahub/maas-api-e2e-api-b-06e51f-6979b974-w92xw" Jul 07 18:29:27.198681 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:27.198648 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a71b337-5f22-43c3-899a-74dbdfad9057-openshift-service-ca\") pod \"maas-api-e2e-api-b-06e51f-6979b974-w92xw\" (UID: \"8a71b337-5f22-43c3-899a-74dbdfad9057\") " pod="opendatahub/maas-api-e2e-api-b-06e51f-6979b974-w92xw" Jul 07 18:29:27.202338 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:27.202304 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a71b337-5f22-43c3-899a-74dbdfad9057-maas-api-tls\") pod \"maas-api-e2e-api-b-06e51f-6979b974-w92xw\" (UID: \"8a71b337-5f22-43c3-899a-74dbdfad9057\") " pod="opendatahub/maas-api-e2e-api-b-06e51f-6979b974-w92xw" Jul 07 18:29:27.208870 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:27.208832 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn2lx\" (UniqueName: \"kubernetes.io/projected/8a71b337-5f22-43c3-899a-74dbdfad9057-kube-api-access-hn2lx\") pod \"maas-api-e2e-api-b-06e51f-6979b974-w92xw\" (UID: \"8a71b337-5f22-43c3-899a-74dbdfad9057\") " pod="opendatahub/maas-api-e2e-api-b-06e51f-6979b974-w92xw" Jul 07 18:29:27.230270 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:27.230239 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:27.271354 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:27.271314 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-06e51f-6979b974-w92xw" Jul 07 18:29:27.439070 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:27.439035 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-06e51f-6979b974-w92xw"] Jul 07 18:29:27.442501 ip-10-0-136-192 kubenswrapper[2583]: W0707 18:29:27.442474 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a71b337_5f22_43c3_899a_74dbdfad9057.slice/crio-a55744e0c62345d78b343592351e62617fad21237bf5e485a87941ff4d05da34 WatchSource:0}: Error finding container a55744e0c62345d78b343592351e62617fad21237bf5e485a87941ff4d05da34: Status 404 returned error can't find the container with id a55744e0c62345d78b343592351e62617fad21237bf5e485a87941ff4d05da34 Jul 07 18:29:27.554702 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:27.554672 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:27.655184 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:27.655090 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:27.779658 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:27.779625 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-06e51f-6979b974-w92xw" event={"ID":"8a71b337-5f22-43c3-899a-74dbdfad9057","Type":"ContainerStarted","Data":"a55744e0c62345d78b343592351e62617fad21237bf5e485a87941ff4d05da34"} Jul 07 18:29:28.786786 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:28.786748 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-06e51f-6979b974-w92xw" event={"ID":"8a71b337-5f22-43c3-899a-74dbdfad9057","Type":"ContainerStarted","Data":"99ddd672c828608903a9530708f6a7eafda96c8df71a20b33b89fa927a57e0f2"} Jul 07 18:29:28.787290 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:28.786822 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-06e51f-6979b974-w92xw" Jul 07 18:29:28.807432 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:28.807384 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-06e51f-6979b974-w92xw" podStartSLOduration=2.485689835 podStartE2EDuration="2.807365382s" totalImagesPullingTime="321.675547ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:29:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:29:28.805300764 +0000 UTC m=+2117.887904468" watchObservedRunningTime="2026-07-07 18:29:28.807365382 +0000 UTC m=+2117.889969085" Jul 07 18:29:30.798095 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:30.798060 2583 generic.go:356] "Generic (PLEG): container finished" podID="d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17" containerID="685b2fa49ec57e7fb598e069765198b7f0b0f3f7f79b825b6594747a906def62" exitCode=0 Jul 07 18:29:30.798549 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:30.798134 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" event={"ID":"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17","Type":"ContainerDied","Data":"685b2fa49ec57e7fb598e069765198b7f0b0f3f7f79b825b6594747a906def62"} Jul 07 18:29:30.798549 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:30.798527 2583 scope.go:117] "RemoveContainer" containerID="685b2fa49ec57e7fb598e069765198b7f0b0f3f7f79b825b6594747a906def62" Jul 07 18:29:30.805108 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:30.800056 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:29:30.805108 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:30.800241 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:29:30.805108 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:30.800336 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:29:31.384469 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:31.384434 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:31.711492 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:31.711322 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:31.808504 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:31.808447 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" event={"ID":"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17","Type":"ContainerStarted","Data":"de5b4abb0c08086ef0dce60e049d147c7e9cd9b93598f3bf1075b5c6c4c2b939"} Jul 07 18:29:32.272123 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:32.272073 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:32.273321 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:32.273293 2583 patch_prober.go:28] interesting pod/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 07 18:29:32.273453 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:32.273345 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" podUID="d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 07 18:29:33.272484 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:33.272444 2583 patch_prober.go:28] interesting pod/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 07 18:29:33.272882 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:33.272504 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" podUID="d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 07 18:29:33.880064 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:33.880010 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:29:34.272398 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:34.272300 2583 patch_prober.go:28] interesting pod/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 07 18:29:34.272398 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:34.272360 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" podUID="d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 07 18:29:34.811148 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:34.811117 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-06e51f-6979b974-w92xw" Jul 07 18:29:35.272267 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:35.272169 2583 patch_prober.go:28] interesting pod/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 07 18:29:35.272267 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:35.272233 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" podUID="d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 07 18:29:36.275477 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:36.275443 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:36.275878 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:36.275731 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:36.276667 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:36.276652 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:39.528537 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:39.528501 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:39.632970 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:39.632940 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:39.815026 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:39.814993 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt"] Jul 07 18:29:39.815273 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:39.815247 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt" podUID="44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0" containerName="maas-api" containerID="cri-o://dacc2ce971e29824e9212fc788eb70e90c0c5d9a02dc1c9732d3f3ee3083bf9c" gracePeriod=30 Jul 07 18:29:39.868936 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:39.868858 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:40.021348 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:40.021309 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:40.081245 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:40.081176 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt" Jul 07 18:29:40.146983 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:40.146950 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0-openshift-service-ca\") pod \"44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0\" (UID: \"44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0\") " Jul 07 18:29:40.147184 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:40.147063 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2q79\" (UniqueName: \"kubernetes.io/projected/44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0-kube-api-access-t2q79\") pod \"44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0\" (UID: \"44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0\") " Jul 07 18:29:40.147184 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:40.147087 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0-maas-api-tls\") pod \"44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0\" (UID: \"44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0\") " Jul 07 18:29:40.147384 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:40.147354 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0" (UID: "44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:29:40.149550 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:40.149504 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0-kube-api-access-t2q79" (OuterVolumeSpecName: "kube-api-access-t2q79") pod "44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0" (UID: "44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0"). InnerVolumeSpecName "kube-api-access-t2q79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:29:40.149712 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:40.149695 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0" (UID: "44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:29:40.248623 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:40.248590 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0-openshift-service-ca\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:40.248623 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:40.248619 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2q79\" (UniqueName: \"kubernetes.io/projected/44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0-kube-api-access-t2q79\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:40.248623 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:40.248630 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0-maas-api-tls\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:40.858276 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:40.858240 2583 generic.go:356] "Generic (PLEG): container finished" podID="44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0" containerID="dacc2ce971e29824e9212fc788eb70e90c0c5d9a02dc1c9732d3f3ee3083bf9c" exitCode=0 Jul 07 18:29:40.858715 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:40.858289 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt" Jul 07 18:29:40.858715 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:40.858328 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt" event={"ID":"44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0","Type":"ContainerDied","Data":"dacc2ce971e29824e9212fc788eb70e90c0c5d9a02dc1c9732d3f3ee3083bf9c"} Jul 07 18:29:40.858715 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:40.858383 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt" event={"ID":"44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0","Type":"ContainerDied","Data":"8e59e6ef70fe11005003f88a389289d18d59a4680fbc68723eb994370386fbc0"} Jul 07 18:29:40.858715 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:40.858405 2583 scope.go:117] "RemoveContainer" containerID="dacc2ce971e29824e9212fc788eb70e90c0c5d9a02dc1c9732d3f3ee3083bf9c" Jul 07 18:29:40.869569 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:40.869523 2583 scope.go:117] "RemoveContainer" containerID="dacc2ce971e29824e9212fc788eb70e90c0c5d9a02dc1c9732d3f3ee3083bf9c" Jul 07 18:29:40.869885 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:29:40.869858 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dacc2ce971e29824e9212fc788eb70e90c0c5d9a02dc1c9732d3f3ee3083bf9c\": container with ID starting with dacc2ce971e29824e9212fc788eb70e90c0c5d9a02dc1c9732d3f3ee3083bf9c not found: ID does not exist" containerID="dacc2ce971e29824e9212fc788eb70e90c0c5d9a02dc1c9732d3f3ee3083bf9c" Jul 07 18:29:40.869989 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:40.869917 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dacc2ce971e29824e9212fc788eb70e90c0c5d9a02dc1c9732d3f3ee3083bf9c"} err="failed to get container status \"dacc2ce971e29824e9212fc788eb70e90c0c5d9a02dc1c9732d3f3ee3083bf9c\": rpc error: code = NotFound desc = could not find container \"dacc2ce971e29824e9212fc788eb70e90c0c5d9a02dc1c9732d3f3ee3083bf9c\": container with ID starting with dacc2ce971e29824e9212fc788eb70e90c0c5d9a02dc1c9732d3f3ee3083bf9c not found: ID does not exist" Jul 07 18:29:40.890349 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:40.890316 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt"] Jul 07 18:29:40.893471 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:40.893445 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-3e6980-67dcf6dddf-dzjkt"] Jul 07 18:29:41.431040 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:41.431000 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0" path="/var/lib/kubelet/pods/44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0/volumes" Jul 07 18:29:41.809098 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:41.809053 2583 patch_prober.go:28] interesting pod/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 07 18:29:41.810367 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:41.809135 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" podUID="4610ded5-b188-438f-9067-0ee96bd33107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 07 18:29:42.874595 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:42.874562 2583 generic.go:356] "Generic (PLEG): container finished" podID="4610ded5-b188-438f-9067-0ee96bd33107" containerID="f474684cd23da2a32898da4edd828761124422856c0675f9355414197d43d52c" exitCode=0 Jul 07 18:29:42.875023 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:42.874637 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" event={"ID":"4610ded5-b188-438f-9067-0ee96bd33107","Type":"ContainerDied","Data":"f474684cd23da2a32898da4edd828761124422856c0675f9355414197d43d52c"} Jul 07 18:29:42.875202 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:42.875183 2583 scope.go:117] "RemoveContainer" containerID="f474684cd23da2a32898da4edd828761124422856c0675f9355414197d43d52c" Jul 07 18:29:42.876368 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:42.876328 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:29:42.876432 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:42.876414 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:29:42.876479 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:42.876464 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:29:43.881300 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:43.881263 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" event={"ID":"4610ded5-b188-438f-9067-0ee96bd33107","Type":"ContainerStarted","Data":"f1eec9f5a5441c5e2d0b3e13542d1ce15b1ad15a82c725380292955688f792a9"} Jul 07 18:29:44.538753 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:44.538712 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:44.539001 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:44.538981 2583 patch_prober.go:28] interesting pod/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 07 18:29:44.539091 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:44.539035 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" podUID="4610ded5-b188-438f-9067-0ee96bd33107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 07 18:29:45.341594 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:45.341552 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:29:45.538277 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:45.538236 2583 patch_prober.go:28] interesting pod/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 07 18:29:45.538491 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:45.538296 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" podUID="4610ded5-b188-438f-9067-0ee96bd33107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 07 18:29:46.353759 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:46.353727 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:46.467180 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:46.467146 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:46.538699 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:46.538662 2583 patch_prober.go:28] interesting pod/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 07 18:29:46.538876 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:46.538716 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" podUID="4610ded5-b188-438f-9067-0ee96bd33107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 07 18:29:46.598655 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:46.598603 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6"] Jul 07 18:29:46.599480 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:46.599436 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" podUID="d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17" containerName="istio-proxy" containerID="cri-o://de5b4abb0c08086ef0dce60e049d147c7e9cd9b93598f3bf1075b5c6c4c2b939" gracePeriod=30 Jul 07 18:29:46.616663 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:46.616610 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:46.835604 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:46.835572 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:47.002997 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:46.999934 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:47.480582 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:47.480546 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:47.538464 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:47.538428 2583 patch_prober.go:28] interesting pod/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 07 18:29:47.538649 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:47.538495 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" podUID="4610ded5-b188-438f-9067-0ee96bd33107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 07 18:29:47.877729 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:47.877693 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:47.971101 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:47.971066 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:48.160380 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:48.160293 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-06e51f-6979b974-w92xw"] Jul 07 18:29:48.160549 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:48.160510 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-06e51f-6979b974-w92xw" podUID="8a71b337-5f22-43c3-899a-74dbdfad9057" containerName="maas-api" containerID="cri-o://99ddd672c828608903a9530708f6a7eafda96c8df71a20b33b89fa927a57e0f2" gracePeriod=30 Jul 07 18:29:48.217054 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:48.217021 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:48.416792 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:48.416718 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-06e51f-6979b974-w92xw" Jul 07 18:29:48.538237 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:48.538202 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hn2lx\" (UniqueName: \"kubernetes.io/projected/8a71b337-5f22-43c3-899a-74dbdfad9057-kube-api-access-hn2lx\") pod \"8a71b337-5f22-43c3-899a-74dbdfad9057\" (UID: \"8a71b337-5f22-43c3-899a-74dbdfad9057\") " Jul 07 18:29:48.538721 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:48.538294 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a71b337-5f22-43c3-899a-74dbdfad9057-openshift-service-ca\") pod \"8a71b337-5f22-43c3-899a-74dbdfad9057\" (UID: \"8a71b337-5f22-43c3-899a-74dbdfad9057\") " Jul 07 18:29:48.538721 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:48.538336 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a71b337-5f22-43c3-899a-74dbdfad9057-maas-api-tls\") pod \"8a71b337-5f22-43c3-899a-74dbdfad9057\" (UID: \"8a71b337-5f22-43c3-899a-74dbdfad9057\") " Jul 07 18:29:48.538721 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:48.538638 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a71b337-5f22-43c3-899a-74dbdfad9057-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8a71b337-5f22-43c3-899a-74dbdfad9057" (UID: "8a71b337-5f22-43c3-899a-74dbdfad9057"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:29:48.540620 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:48.540584 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a71b337-5f22-43c3-899a-74dbdfad9057-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8a71b337-5f22-43c3-899a-74dbdfad9057" (UID: "8a71b337-5f22-43c3-899a-74dbdfad9057"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:29:48.540743 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:48.540688 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a71b337-5f22-43c3-899a-74dbdfad9057-kube-api-access-hn2lx" (OuterVolumeSpecName: "kube-api-access-hn2lx") pod "8a71b337-5f22-43c3-899a-74dbdfad9057" (UID: "8a71b337-5f22-43c3-899a-74dbdfad9057"). InnerVolumeSpecName "kube-api-access-hn2lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:29:48.541738 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:48.541717 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:48.542124 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:48.542100 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:48.542866 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:48.542849 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:48.639544 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:48.639504 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn2lx\" (UniqueName: \"kubernetes.io/projected/8a71b337-5f22-43c3-899a-74dbdfad9057-kube-api-access-hn2lx\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:48.639544 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:48.639539 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a71b337-5f22-43c3-899a-74dbdfad9057-openshift-service-ca\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:48.639544 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:48.639555 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a71b337-5f22-43c3-899a-74dbdfad9057-maas-api-tls\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:48.915349 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:48.915310 2583 generic.go:356] "Generic (PLEG): container finished" podID="8a71b337-5f22-43c3-899a-74dbdfad9057" containerID="99ddd672c828608903a9530708f6a7eafda96c8df71a20b33b89fa927a57e0f2" exitCode=0 Jul 07 18:29:48.915562 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:48.915362 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-06e51f-6979b974-w92xw" Jul 07 18:29:48.915562 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:48.915406 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-06e51f-6979b974-w92xw" event={"ID":"8a71b337-5f22-43c3-899a-74dbdfad9057","Type":"ContainerDied","Data":"99ddd672c828608903a9530708f6a7eafda96c8df71a20b33b89fa927a57e0f2"} Jul 07 18:29:48.915562 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:48.915468 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-06e51f-6979b974-w92xw" event={"ID":"8a71b337-5f22-43c3-899a-74dbdfad9057","Type":"ContainerDied","Data":"a55744e0c62345d78b343592351e62617fad21237bf5e485a87941ff4d05da34"} Jul 07 18:29:48.915562 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:48.915498 2583 scope.go:117] "RemoveContainer" containerID="99ddd672c828608903a9530708f6a7eafda96c8df71a20b33b89fa927a57e0f2" Jul 07 18:29:48.925267 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:48.925247 2583 scope.go:117] "RemoveContainer" containerID="99ddd672c828608903a9530708f6a7eafda96c8df71a20b33b89fa927a57e0f2" Jul 07 18:29:48.925577 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:29:48.925555 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99ddd672c828608903a9530708f6a7eafda96c8df71a20b33b89fa927a57e0f2\": container with ID starting with 99ddd672c828608903a9530708f6a7eafda96c8df71a20b33b89fa927a57e0f2 not found: ID does not exist" containerID="99ddd672c828608903a9530708f6a7eafda96c8df71a20b33b89fa927a57e0f2" Jul 07 18:29:48.925655 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:48.925584 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99ddd672c828608903a9530708f6a7eafda96c8df71a20b33b89fa927a57e0f2"} err="failed to get container status \"99ddd672c828608903a9530708f6a7eafda96c8df71a20b33b89fa927a57e0f2\": rpc error: code = NotFound desc = could not find container \"99ddd672c828608903a9530708f6a7eafda96c8df71a20b33b89fa927a57e0f2\": container with ID starting with 99ddd672c828608903a9530708f6a7eafda96c8df71a20b33b89fa927a57e0f2 not found: ID does not exist" Jul 07 18:29:48.936865 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:48.936832 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-06e51f-6979b974-w92xw"] Jul 07 18:29:48.940758 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:48.940725 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-06e51f-6979b974-w92xw"] Jul 07 18:29:49.429381 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:49.429346 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a71b337-5f22-43c3-899a-74dbdfad9057" path="/var/lib/kubelet/pods/8a71b337-5f22-43c3-899a-74dbdfad9057/volumes" Jul 07 18:29:51.278443 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.278397 2583 patch_prober.go:28] interesting pod/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 18:29:51.278830 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.278455 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" podUID="d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 18:29:51.896216 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.896185 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:51.933948 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.933893 2583 generic.go:356] "Generic (PLEG): container finished" podID="d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17" containerID="de5b4abb0c08086ef0dce60e049d147c7e9cd9b93598f3bf1075b5c6c4c2b939" exitCode=0 Jul 07 18:29:51.933948 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.933938 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" event={"ID":"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17","Type":"ContainerDied","Data":"de5b4abb0c08086ef0dce60e049d147c7e9cd9b93598f3bf1075b5c6c4c2b939"} Jul 07 18:29:51.934203 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.934002 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" event={"ID":"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17","Type":"ContainerDied","Data":"b689eac2be4cc53fd7481ee50f3c4f77ad3a5a59d46f3641924d599fb3a20d9d"} Jul 07 18:29:51.934203 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.934025 2583 scope.go:117] "RemoveContainer" containerID="de5b4abb0c08086ef0dce60e049d147c7e9cd9b93598f3bf1075b5c6c4c2b939" Jul 07 18:29:51.934203 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.933953 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6" Jul 07 18:29:51.943417 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.943392 2583 scope.go:117] "RemoveContainer" containerID="685b2fa49ec57e7fb598e069765198b7f0b0f3f7f79b825b6594747a906def62" Jul 07 18:29:51.953587 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.953545 2583 scope.go:117] "RemoveContainer" containerID="de5b4abb0c08086ef0dce60e049d147c7e9cd9b93598f3bf1075b5c6c4c2b939" Jul 07 18:29:51.953859 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:29:51.953829 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de5b4abb0c08086ef0dce60e049d147c7e9cd9b93598f3bf1075b5c6c4c2b939\": container with ID starting with de5b4abb0c08086ef0dce60e049d147c7e9cd9b93598f3bf1075b5c6c4c2b939 not found: ID does not exist" containerID="de5b4abb0c08086ef0dce60e049d147c7e9cd9b93598f3bf1075b5c6c4c2b939" Jul 07 18:29:51.953991 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.953872 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5b4abb0c08086ef0dce60e049d147c7e9cd9b93598f3bf1075b5c6c4c2b939"} err="failed to get container status \"de5b4abb0c08086ef0dce60e049d147c7e9cd9b93598f3bf1075b5c6c4c2b939\": rpc error: code = NotFound desc = could not find container \"de5b4abb0c08086ef0dce60e049d147c7e9cd9b93598f3bf1075b5c6c4c2b939\": container with ID starting with de5b4abb0c08086ef0dce60e049d147c7e9cd9b93598f3bf1075b5c6c4c2b939 not found: ID does not exist" Jul 07 18:29:51.953991 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.953894 2583 scope.go:117] "RemoveContainer" containerID="685b2fa49ec57e7fb598e069765198b7f0b0f3f7f79b825b6594747a906def62" Jul 07 18:29:51.954253 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:29:51.954220 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"685b2fa49ec57e7fb598e069765198b7f0b0f3f7f79b825b6594747a906def62\": container with ID starting with 685b2fa49ec57e7fb598e069765198b7f0b0f3f7f79b825b6594747a906def62 not found: ID does not exist" containerID="685b2fa49ec57e7fb598e069765198b7f0b0f3f7f79b825b6594747a906def62" Jul 07 18:29:51.954380 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.954254 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"685b2fa49ec57e7fb598e069765198b7f0b0f3f7f79b825b6594747a906def62"} err="failed to get container status \"685b2fa49ec57e7fb598e069765198b7f0b0f3f7f79b825b6594747a906def62\": rpc error: code = NotFound desc = could not find container \"685b2fa49ec57e7fb598e069765198b7f0b0f3f7f79b825b6594747a906def62\": container with ID starting with 685b2fa49ec57e7fb598e069765198b7f0b0f3f7f79b825b6594747a906def62 not found: ID does not exist" Jul 07 18:29:51.976791 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.976751 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpf95\" (UniqueName: \"kubernetes.io/projected/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-kube-api-access-mpf95\") pod \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " Jul 07 18:29:51.977016 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.976811 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-istio-podinfo\") pod \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " Jul 07 18:29:51.977016 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.976871 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-istio-token\") pod \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " Jul 07 18:29:51.977016 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.976951 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-istio-data\") pod \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " Jul 07 18:29:51.977190 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.977123 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-istio-envoy\") pod \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " Jul 07 18:29:51.977190 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.977175 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-workload-certs\") pod \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " Jul 07 18:29:51.977303 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.977200 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-credential-socket\") pod \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " Jul 07 18:29:51.977303 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.977245 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-workload-socket\") pod \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " Jul 07 18:29:51.977303 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.977285 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-istiod-ca-cert\") pod \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\" (UID: \"d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17\") " Jul 07 18:29:51.977493 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.977445 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17" (UID: "d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:29:51.977956 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.977856 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17" (UID: "d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:29:51.977956 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.977887 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-workload-certs\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:51.978239 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.978209 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17" (UID: "d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:29:51.978317 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.978217 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-istio-data" (OuterVolumeSpecName: "istio-data") pod "d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17" (UID: "d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:29:51.978507 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.978479 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17" (UID: "d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:29:51.979460 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.979436 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17" (UID: "d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 18:29:51.979460 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.979442 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-kube-api-access-mpf95" (OuterVolumeSpecName: "kube-api-access-mpf95") pod "d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17" (UID: "d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17"). InnerVolumeSpecName "kube-api-access-mpf95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:29:51.979589 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.979457 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-istio-token" (OuterVolumeSpecName: "istio-token") pod "d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17" (UID: "d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:29:51.979589 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:51.979465 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17" (UID: "d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:29:52.079283 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:52.079243 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-istio-envoy\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:52.079283 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:52.079276 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-credential-socket\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:52.079283 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:52.079286 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-workload-socket\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:52.079537 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:52.079294 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-istiod-ca-cert\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:52.079537 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:52.079305 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpf95\" (UniqueName: \"kubernetes.io/projected/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-kube-api-access-mpf95\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:52.079537 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:52.079313 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-istio-podinfo\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:52.079537 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:52.079321 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-istio-token\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:52.079537 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:52.079330 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17-istio-data\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:52.256895 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:52.256856 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6"] Jul 07 18:29:52.263465 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:52.263434 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-3e6980-openshift-default-756ffb9f57-bkrx6"] Jul 07 18:29:53.430120 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:53.430080 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17" path="/var/lib/kubelet/pods/d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17/volumes" Jul 07 18:29:53.690641 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:53.690548 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:53.752010 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:53.751972 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:54.008535 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:54.008442 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx"] Jul 07 18:29:54.008698 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:54.008655 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" podUID="4610ded5-b188-438f-9067-0ee96bd33107" containerName="istio-proxy" containerID="cri-o://f1eec9f5a5441c5e2d0b3e13542d1ce15b1ad15a82c725380292955688f792a9" gracePeriod=30 Jul 07 18:29:54.047889 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:54.047852 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:54.282016 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:54.281919 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:54.603079 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:54.603034 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:54.940486 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:54.940401 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s"] Jul 07 18:29:54.940776 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:54.940764 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a71b337-5f22-43c3-899a-74dbdfad9057" containerName="maas-api" Jul 07 18:29:54.940824 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:54.940778 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a71b337-5f22-43c3-899a-74dbdfad9057" containerName="maas-api" Jul 07 18:29:54.940824 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:54.940789 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17" containerName="istio-proxy" Jul 07 18:29:54.940824 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:54.940795 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17" containerName="istio-proxy" Jul 07 18:29:54.940824 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:54.940813 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0" containerName="maas-api" Jul 07 18:29:54.940824 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:54.940819 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0" containerName="maas-api" Jul 07 18:29:54.940993 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:54.940874 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="44d0c9d1-8ae7-4f4d-a971-474bc6f22dd0" containerName="maas-api" Jul 07 18:29:54.940993 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:54.940882 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17" containerName="istio-proxy" Jul 07 18:29:54.940993 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:54.940891 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17" containerName="istio-proxy" Jul 07 18:29:54.940993 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:54.940911 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a71b337-5f22-43c3-899a-74dbdfad9057" containerName="maas-api" Jul 07 18:29:54.945481 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:54.945458 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:54.948159 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:54.948136 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-2b5160-openshift-default-dockercfg-qcp9n\"" Jul 07 18:29:54.956357 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:54.956310 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:54.956484 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:54.956391 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s"] Jul 07 18:29:55.007305 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.007265 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/678edfb4-f4ee-41e4-af93-f2a741db1657-credential-socket\") pod \"e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:55.007493 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.007325 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/678edfb4-f4ee-41e4-af93-f2a741db1657-workload-certs\") pod \"e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:55.007493 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.007404 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/678edfb4-f4ee-41e4-af93-f2a741db1657-istio-data\") pod \"e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:55.007493 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.007443 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/678edfb4-f4ee-41e4-af93-f2a741db1657-istio-token\") pod \"e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:55.007493 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.007487 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/678edfb4-f4ee-41e4-af93-f2a741db1657-workload-socket\") pod \"e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:55.007678 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.007532 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/678edfb4-f4ee-41e4-af93-f2a741db1657-istiod-ca-cert\") pod \"e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:55.007678 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.007551 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/678edfb4-f4ee-41e4-af93-f2a741db1657-istio-podinfo\") pod \"e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:55.007678 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.007585 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/678edfb4-f4ee-41e4-af93-f2a741db1657-istio-envoy\") pod \"e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:55.007678 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.007622 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dbqg\" (UniqueName: \"kubernetes.io/projected/678edfb4-f4ee-41e4-af93-f2a741db1657-kube-api-access-2dbqg\") pod \"e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:55.057314 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.057279 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:55.108712 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.108677 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/678edfb4-f4ee-41e4-af93-f2a741db1657-istio-data\") pod \"e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:55.108712 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.108720 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/678edfb4-f4ee-41e4-af93-f2a741db1657-istio-token\") pod \"e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:55.109017 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.108787 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/678edfb4-f4ee-41e4-af93-f2a741db1657-workload-socket\") pod \"e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:55.109017 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.108827 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/678edfb4-f4ee-41e4-af93-f2a741db1657-istiod-ca-cert\") pod \"e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:55.109017 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.108856 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/678edfb4-f4ee-41e4-af93-f2a741db1657-istio-podinfo\") pod \"e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:55.109017 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.108932 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/678edfb4-f4ee-41e4-af93-f2a741db1657-istio-envoy\") pod \"e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:55.109017 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.108962 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dbqg\" (UniqueName: \"kubernetes.io/projected/678edfb4-f4ee-41e4-af93-f2a741db1657-kube-api-access-2dbqg\") pod \"e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:55.109017 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.109011 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/678edfb4-f4ee-41e4-af93-f2a741db1657-credential-socket\") pod \"e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:55.109349 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.109058 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/678edfb4-f4ee-41e4-af93-f2a741db1657-workload-certs\") pod \"e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:55.109434 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.109404 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/678edfb4-f4ee-41e4-af93-f2a741db1657-istio-data\") pod \"e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:55.109824 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.109496 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/678edfb4-f4ee-41e4-af93-f2a741db1657-credential-socket\") pod \"e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:55.109824 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.109680 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/678edfb4-f4ee-41e4-af93-f2a741db1657-workload-socket\") pod \"e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:55.109824 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.109779 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/678edfb4-f4ee-41e4-af93-f2a741db1657-workload-certs\") pod \"e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:55.110225 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.110201 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/678edfb4-f4ee-41e4-af93-f2a741db1657-istiod-ca-cert\") pod \"e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:55.112306 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.112281 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/678edfb4-f4ee-41e4-af93-f2a741db1657-istio-envoy\") pod \"e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:55.112724 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.112703 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/678edfb4-f4ee-41e4-af93-f2a741db1657-istio-podinfo\") pod \"e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:55.121007 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.120947 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/678edfb4-f4ee-41e4-af93-f2a741db1657-istio-token\") pod \"e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:55.121007 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.120984 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dbqg\" (UniqueName: \"kubernetes.io/projected/678edfb4-f4ee-41e4-af93-f2a741db1657-kube-api-access-2dbqg\") pod \"e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:55.258960 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.258848 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:55.419108 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.419076 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s"] Jul 07 18:29:55.422487 ip-10-0-136-192 kubenswrapper[2583]: W0707 18:29:55.422460 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod678edfb4_f4ee_41e4_af93_f2a741db1657.slice/crio-72224ed065df7352a5ab61d879819a4e2b9cf26482b83135b1c0845ea9b40847 WatchSource:0}: Error finding container 72224ed065df7352a5ab61d879819a4e2b9cf26482b83135b1c0845ea9b40847: Status 404 returned error can't find the container with id 72224ed065df7352a5ab61d879819a4e2b9cf26482b83135b1c0845ea9b40847 Jul 07 18:29:55.424824 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.424789 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:29:55.424946 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.424894 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:29:55.425015 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.424961 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:29:55.959596 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.959566 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" event={"ID":"678edfb4-f4ee-41e4-af93-f2a741db1657","Type":"ContainerStarted","Data":"0246789ae6357092cdc749c79c4d386739baba33ab3a6e810ee42d122b099dc6"} Jul 07 18:29:55.960022 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.959615 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" event={"ID":"678edfb4-f4ee-41e4-af93-f2a741db1657","Type":"ContainerStarted","Data":"72224ed065df7352a5ab61d879819a4e2b9cf26482b83135b1c0845ea9b40847"} Jul 07 18:29:55.983035 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:55.982980 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" podStartSLOduration=1.9829591899999999 podStartE2EDuration="1.98295919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:29:55.981764892 +0000 UTC m=+2145.064368600" watchObservedRunningTime="2026-07-07 18:29:55.98295919 +0000 UTC m=+2145.065562885" Jul 07 18:29:56.260027 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:56.259931 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:56.265149 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:56.265121 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:56.965309 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:56.965270 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:56.966300 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:56.966281 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:29:57.945404 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:57.945365 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:58.015656 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:58.015621 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-2b5160-58f958d857-rwj96"] Jul 07 18:29:58.016175 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:58.016041 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17" containerName="istio-proxy" Jul 07 18:29:58.016175 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:58.016053 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d459071e-6ede-4bf7-ac3b-b5c0fb0a0d17" containerName="istio-proxy" Jul 07 18:29:58.019915 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:58.019876 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-2b5160-58f958d857-rwj96" Jul 07 18:29:58.022366 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:58.022341 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-2b5160-serving-cert\"" Jul 07 18:29:58.022525 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:58.022409 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-jnv2d\"" Jul 07 18:29:58.022525 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:58.022495 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:58.030720 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:58.030687 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-2b5160-58f958d857-rwj96"] Jul 07 18:29:58.099471 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:58.099436 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:58.142761 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:58.142722 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d12baac8-ff09-4c0c-879a-0914c5dcaef3-maas-api-tls\") pod \"maas-api-e2e-shared-a-2b5160-58f958d857-rwj96\" (UID: \"d12baac8-ff09-4c0c-879a-0914c5dcaef3\") " pod="opendatahub/maas-api-e2e-shared-a-2b5160-58f958d857-rwj96" Jul 07 18:29:58.142985 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:58.142842 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jnmr\" (UniqueName: \"kubernetes.io/projected/d12baac8-ff09-4c0c-879a-0914c5dcaef3-kube-api-access-7jnmr\") pod \"maas-api-e2e-shared-a-2b5160-58f958d857-rwj96\" (UID: \"d12baac8-ff09-4c0c-879a-0914c5dcaef3\") " pod="opendatahub/maas-api-e2e-shared-a-2b5160-58f958d857-rwj96" Jul 07 18:29:58.142985 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:58.142867 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d12baac8-ff09-4c0c-879a-0914c5dcaef3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-2b5160-58f958d857-rwj96\" (UID: \"d12baac8-ff09-4c0c-879a-0914c5dcaef3\") " pod="opendatahub/maas-api-e2e-shared-a-2b5160-58f958d857-rwj96" Jul 07 18:29:58.151036 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:58.151004 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:29:58.199889 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:58.199811 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:58.244046 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:58.244006 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jnmr\" (UniqueName: \"kubernetes.io/projected/d12baac8-ff09-4c0c-879a-0914c5dcaef3-kube-api-access-7jnmr\") pod \"maas-api-e2e-shared-a-2b5160-58f958d857-rwj96\" (UID: \"d12baac8-ff09-4c0c-879a-0914c5dcaef3\") " pod="opendatahub/maas-api-e2e-shared-a-2b5160-58f958d857-rwj96" Jul 07 18:29:58.244046 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:58.244049 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d12baac8-ff09-4c0c-879a-0914c5dcaef3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-2b5160-58f958d857-rwj96\" (UID: \"d12baac8-ff09-4c0c-879a-0914c5dcaef3\") " pod="opendatahub/maas-api-e2e-shared-a-2b5160-58f958d857-rwj96" Jul 07 18:29:58.244345 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:58.244126 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d12baac8-ff09-4c0c-879a-0914c5dcaef3-maas-api-tls\") pod \"maas-api-e2e-shared-a-2b5160-58f958d857-rwj96\" (UID: \"d12baac8-ff09-4c0c-879a-0914c5dcaef3\") " pod="opendatahub/maas-api-e2e-shared-a-2b5160-58f958d857-rwj96" Jul 07 18:29:58.244868 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:58.244841 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d12baac8-ff09-4c0c-879a-0914c5dcaef3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-2b5160-58f958d857-rwj96\" (UID: \"d12baac8-ff09-4c0c-879a-0914c5dcaef3\") " pod="opendatahub/maas-api-e2e-shared-a-2b5160-58f958d857-rwj96" Jul 07 18:29:58.246877 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:58.246849 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d12baac8-ff09-4c0c-879a-0914c5dcaef3-maas-api-tls\") pod \"maas-api-e2e-shared-a-2b5160-58f958d857-rwj96\" (UID: \"d12baac8-ff09-4c0c-879a-0914c5dcaef3\") " pod="opendatahub/maas-api-e2e-shared-a-2b5160-58f958d857-rwj96" Jul 07 18:29:58.255426 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:58.255401 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jnmr\" (UniqueName: \"kubernetes.io/projected/d12baac8-ff09-4c0c-879a-0914c5dcaef3-kube-api-access-7jnmr\") pod \"maas-api-e2e-shared-a-2b5160-58f958d857-rwj96\" (UID: \"d12baac8-ff09-4c0c-879a-0914c5dcaef3\") " pod="opendatahub/maas-api-e2e-shared-a-2b5160-58f958d857-rwj96" Jul 07 18:29:58.335058 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:58.335022 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-2b5160-58f958d857-rwj96" Jul 07 18:29:58.483163 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:58.482978 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-2b5160-58f958d857-rwj96"] Jul 07 18:29:58.485583 ip-10-0-136-192 kubenswrapper[2583]: W0707 18:29:58.485552 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12baac8_ff09_4c0c_879a_0914c5dcaef3.slice/crio-eb4cd0b34f123cff0117984bcde0aa3677acdd91f1689bb0e8b6869b3e21857e WatchSource:0}: Error finding container eb4cd0b34f123cff0117984bcde0aa3677acdd91f1689bb0e8b6869b3e21857e: Status 404 returned error can't find the container with id eb4cd0b34f123cff0117984bcde0aa3677acdd91f1689bb0e8b6869b3e21857e Jul 07 18:29:58.516345 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:58.516303 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:58.617102 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:58.617072 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:29:58.977648 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:58.977613 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-2b5160-58f958d857-rwj96" event={"ID":"d12baac8-ff09-4c0c-879a-0914c5dcaef3","Type":"ContainerStarted","Data":"70f5b11893c1fb9aa8395e51098301dea3d3ec35f77f8a3b9455e9f0494596f5"} Jul 07 18:29:58.977826 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:58.977665 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-2b5160-58f958d857-rwj96" event={"ID":"d12baac8-ff09-4c0c-879a-0914c5dcaef3","Type":"ContainerStarted","Data":"eb4cd0b34f123cff0117984bcde0aa3677acdd91f1689bb0e8b6869b3e21857e"} Jul 07 18:29:58.977868 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:58.977822 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-2b5160-58f958d857-rwj96" Jul 07 18:29:58.996926 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:58.996837 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-2b5160-58f958d857-rwj96" podStartSLOduration=1.6967111780000002 podStartE2EDuration="1.996815816s" totalImagesPullingTime="300.104638ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:29:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:29:58.996100457 +0000 UTC m=+2148.078704161" watchObservedRunningTime="2026-07-07 18:29:58.996815816 +0000 UTC m=+2148.079419523" Jul 07 18:29:59.282475 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.282441 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:59.356139 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.356102 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4610ded5-b188-438f-9067-0ee96bd33107-istio-token\") pod \"4610ded5-b188-438f-9067-0ee96bd33107\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " Jul 07 18:29:59.356298 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.356153 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4610ded5-b188-438f-9067-0ee96bd33107-workload-certs\") pod \"4610ded5-b188-438f-9067-0ee96bd33107\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " Jul 07 18:29:59.356298 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.356191 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4610ded5-b188-438f-9067-0ee96bd33107-istio-envoy\") pod \"4610ded5-b188-438f-9067-0ee96bd33107\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " Jul 07 18:29:59.356298 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.356271 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4610ded5-b188-438f-9067-0ee96bd33107-istio-podinfo\") pod \"4610ded5-b188-438f-9067-0ee96bd33107\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " Jul 07 18:29:59.356494 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.356329 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4610ded5-b188-438f-9067-0ee96bd33107-istiod-ca-cert\") pod \"4610ded5-b188-438f-9067-0ee96bd33107\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " Jul 07 18:29:59.356494 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.356354 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4610ded5-b188-438f-9067-0ee96bd33107-workload-socket\") pod \"4610ded5-b188-438f-9067-0ee96bd33107\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " Jul 07 18:29:59.356494 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.356408 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79r7m\" (UniqueName: \"kubernetes.io/projected/4610ded5-b188-438f-9067-0ee96bd33107-kube-api-access-79r7m\") pod \"4610ded5-b188-438f-9067-0ee96bd33107\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " Jul 07 18:29:59.356494 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.356438 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4610ded5-b188-438f-9067-0ee96bd33107-credential-socket\") pod \"4610ded5-b188-438f-9067-0ee96bd33107\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " Jul 07 18:29:59.356494 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.356461 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4610ded5-b188-438f-9067-0ee96bd33107-istio-data\") pod \"4610ded5-b188-438f-9067-0ee96bd33107\" (UID: \"4610ded5-b188-438f-9067-0ee96bd33107\") " Jul 07 18:29:59.356814 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.356669 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4610ded5-b188-438f-9067-0ee96bd33107-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4610ded5-b188-438f-9067-0ee96bd33107" (UID: "4610ded5-b188-438f-9067-0ee96bd33107"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:29:59.356814 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.356698 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4610ded5-b188-438f-9067-0ee96bd33107-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4610ded5-b188-438f-9067-0ee96bd33107" (UID: "4610ded5-b188-438f-9067-0ee96bd33107"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:29:59.356814 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.356781 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4610ded5-b188-438f-9067-0ee96bd33107-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4610ded5-b188-438f-9067-0ee96bd33107" (UID: "4610ded5-b188-438f-9067-0ee96bd33107"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:29:59.357175 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.356930 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4610ded5-b188-438f-9067-0ee96bd33107-istiod-ca-cert\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:59.357175 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.356947 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4610ded5-b188-438f-9067-0ee96bd33107-workload-socket\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:59.357175 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.356965 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4610ded5-b188-438f-9067-0ee96bd33107-workload-certs\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:59.357175 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.356984 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4610ded5-b188-438f-9067-0ee96bd33107-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4610ded5-b188-438f-9067-0ee96bd33107" (UID: "4610ded5-b188-438f-9067-0ee96bd33107"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:29:59.357817 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.357785 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4610ded5-b188-438f-9067-0ee96bd33107-istio-data" (OuterVolumeSpecName: "istio-data") pod "4610ded5-b188-438f-9067-0ee96bd33107" (UID: "4610ded5-b188-438f-9067-0ee96bd33107"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:29:59.358641 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.358614 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4610ded5-b188-438f-9067-0ee96bd33107-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4610ded5-b188-438f-9067-0ee96bd33107" (UID: "4610ded5-b188-438f-9067-0ee96bd33107"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 18:29:59.358762 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.358704 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4610ded5-b188-438f-9067-0ee96bd33107-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4610ded5-b188-438f-9067-0ee96bd33107" (UID: "4610ded5-b188-438f-9067-0ee96bd33107"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:29:59.358762 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.358732 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4610ded5-b188-438f-9067-0ee96bd33107-istio-token" (OuterVolumeSpecName: "istio-token") pod "4610ded5-b188-438f-9067-0ee96bd33107" (UID: "4610ded5-b188-438f-9067-0ee96bd33107"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:29:59.358839 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.358781 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4610ded5-b188-438f-9067-0ee96bd33107-kube-api-access-79r7m" (OuterVolumeSpecName: "kube-api-access-79r7m") pod "4610ded5-b188-438f-9067-0ee96bd33107" (UID: "4610ded5-b188-438f-9067-0ee96bd33107"). InnerVolumeSpecName "kube-api-access-79r7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:29:59.457805 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.457763 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79r7m\" (UniqueName: \"kubernetes.io/projected/4610ded5-b188-438f-9067-0ee96bd33107-kube-api-access-79r7m\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:59.457805 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.457797 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4610ded5-b188-438f-9067-0ee96bd33107-credential-socket\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:59.457805 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.457810 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4610ded5-b188-438f-9067-0ee96bd33107-istio-data\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:59.458076 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.457822 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4610ded5-b188-438f-9067-0ee96bd33107-istio-token\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:59.458076 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.457833 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4610ded5-b188-438f-9067-0ee96bd33107-istio-envoy\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:59.458076 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.457845 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4610ded5-b188-438f-9067-0ee96bd33107-istio-podinfo\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:29:59.984705 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.984665 2583 generic.go:356] "Generic (PLEG): container finished" podID="4610ded5-b188-438f-9067-0ee96bd33107" containerID="f1eec9f5a5441c5e2d0b3e13542d1ce15b1ad15a82c725380292955688f792a9" exitCode=0 Jul 07 18:29:59.984875 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.984721 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" Jul 07 18:29:59.984875 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.984748 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" event={"ID":"4610ded5-b188-438f-9067-0ee96bd33107","Type":"ContainerDied","Data":"f1eec9f5a5441c5e2d0b3e13542d1ce15b1ad15a82c725380292955688f792a9"} Jul 07 18:29:59.984875 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.984799 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx" event={"ID":"4610ded5-b188-438f-9067-0ee96bd33107","Type":"ContainerDied","Data":"b86b45ccbf4f2201713df5ecaa7b999e9b072a58416c8494b871f6d4613e4e90"} Jul 07 18:29:59.984875 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.984822 2583 scope.go:117] "RemoveContainer" containerID="f1eec9f5a5441c5e2d0b3e13542d1ce15b1ad15a82c725380292955688f792a9" Jul 07 18:29:59.994345 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:29:59.994321 2583 scope.go:117] "RemoveContainer" containerID="f474684cd23da2a32898da4edd828761124422856c0675f9355414197d43d52c" Jul 07 18:30:00.003576 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:00.003553 2583 scope.go:117] "RemoveContainer" containerID="f1eec9f5a5441c5e2d0b3e13542d1ce15b1ad15a82c725380292955688f792a9" Jul 07 18:30:00.003916 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:30:00.003873 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1eec9f5a5441c5e2d0b3e13542d1ce15b1ad15a82c725380292955688f792a9\": container with ID starting with f1eec9f5a5441c5e2d0b3e13542d1ce15b1ad15a82c725380292955688f792a9 not found: ID does not exist" containerID="f1eec9f5a5441c5e2d0b3e13542d1ce15b1ad15a82c725380292955688f792a9" Jul 07 18:30:00.004017 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:00.003938 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1eec9f5a5441c5e2d0b3e13542d1ce15b1ad15a82c725380292955688f792a9"} err="failed to get container status \"f1eec9f5a5441c5e2d0b3e13542d1ce15b1ad15a82c725380292955688f792a9\": rpc error: code = NotFound desc = could not find container \"f1eec9f5a5441c5e2d0b3e13542d1ce15b1ad15a82c725380292955688f792a9\": container with ID starting with f1eec9f5a5441c5e2d0b3e13542d1ce15b1ad15a82c725380292955688f792a9 not found: ID does not exist" Jul 07 18:30:00.004017 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:00.003961 2583 scope.go:117] "RemoveContainer" containerID="f474684cd23da2a32898da4edd828761124422856c0675f9355414197d43d52c" Jul 07 18:30:00.004273 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:30:00.004247 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f474684cd23da2a32898da4edd828761124422856c0675f9355414197d43d52c\": container with ID starting with f474684cd23da2a32898da4edd828761124422856c0675f9355414197d43d52c not found: ID does not exist" containerID="f474684cd23da2a32898da4edd828761124422856c0675f9355414197d43d52c" Jul 07 18:30:00.004345 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:00.004271 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f474684cd23da2a32898da4edd828761124422856c0675f9355414197d43d52c"} err="failed to get container status \"f474684cd23da2a32898da4edd828761124422856c0675f9355414197d43d52c\": rpc error: code = NotFound desc = could not find container \"f474684cd23da2a32898da4edd828761124422856c0675f9355414197d43d52c\": container with ID starting with f474684cd23da2a32898da4edd828761124422856c0675f9355414197d43d52c not found: ID does not exist" Jul 07 18:30:00.005066 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:00.005044 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx"] Jul 07 18:30:00.007596 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:00.007575 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-06e51f-openshift-default-7f9954bc9f-rsxsx"] Jul 07 18:30:01.429129 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:01.429049 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4610ded5-b188-438f-9067-0ee96bd33107" path="/var/lib/kubelet/pods/4610ded5-b188-438f-9067-0ee96bd33107/volumes" Jul 07 18:30:02.621486 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:02.621457 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:02.799805 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:02.799777 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:03.165983 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.165947 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25"] Jul 07 18:30:03.166349 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.166335 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4610ded5-b188-438f-9067-0ee96bd33107" containerName="istio-proxy" Jul 07 18:30:03.166406 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.166352 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4610ded5-b188-438f-9067-0ee96bd33107" containerName="istio-proxy" Jul 07 18:30:03.166406 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.166363 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4610ded5-b188-438f-9067-0ee96bd33107" containerName="istio-proxy" Jul 07 18:30:03.166406 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.166368 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4610ded5-b188-438f-9067-0ee96bd33107" containerName="istio-proxy" Jul 07 18:30:03.166499 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.166446 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4610ded5-b188-438f-9067-0ee96bd33107" containerName="istio-proxy" Jul 07 18:30:03.166499 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.166454 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4610ded5-b188-438f-9067-0ee96bd33107" containerName="istio-proxy" Jul 07 18:30:03.170741 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.170719 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:03.174664 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.174640 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-200b05-openshift-default-dockercfg-hlh5c\"" Jul 07 18:30:03.193988 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.193945 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25"] Jul 07 18:30:03.232225 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.232167 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:03.293381 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.293345 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20ddc125-fd2e-408a-9750-e817f2d9bf6c-istio-envoy\") pod \"e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:03.293381 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.293384 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20ddc125-fd2e-408a-9750-e817f2d9bf6c-istio-podinfo\") pod \"e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:03.293670 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.293428 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqbws\" (UniqueName: \"kubernetes.io/projected/20ddc125-fd2e-408a-9750-e817f2d9bf6c-kube-api-access-jqbws\") pod \"e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:03.293670 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.293484 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ddc125-fd2e-408a-9750-e817f2d9bf6c-credential-socket\") pod \"e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:03.293670 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.293500 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20ddc125-fd2e-408a-9750-e817f2d9bf6c-istio-token\") pod \"e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:03.293670 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.293527 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20ddc125-fd2e-408a-9750-e817f2d9bf6c-workload-certs\") pod \"e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:03.293670 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.293560 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20ddc125-fd2e-408a-9750-e817f2d9bf6c-istio-data\") pod \"e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:03.293890 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.293672 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ddc125-fd2e-408a-9750-e817f2d9bf6c-workload-socket\") pod \"e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:03.293890 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.293730 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20ddc125-fd2e-408a-9750-e817f2d9bf6c-istiod-ca-cert\") pod \"e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:03.321730 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.321693 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:03.394287 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.394246 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ddc125-fd2e-408a-9750-e817f2d9bf6c-credential-socket\") pod \"e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:03.394287 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.394287 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20ddc125-fd2e-408a-9750-e817f2d9bf6c-istio-token\") pod \"e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:03.394557 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.394327 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20ddc125-fd2e-408a-9750-e817f2d9bf6c-workload-certs\") pod \"e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:03.394557 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.394359 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20ddc125-fd2e-408a-9750-e817f2d9bf6c-istio-data\") pod \"e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:03.394557 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.394409 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ddc125-fd2e-408a-9750-e817f2d9bf6c-workload-socket\") pod \"e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:03.394557 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.394449 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20ddc125-fd2e-408a-9750-e817f2d9bf6c-istiod-ca-cert\") pod \"e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:03.394557 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.394478 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20ddc125-fd2e-408a-9750-e817f2d9bf6c-istio-envoy\") pod \"e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:03.394557 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.394504 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20ddc125-fd2e-408a-9750-e817f2d9bf6c-istio-podinfo\") pod \"e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:03.394875 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.394576 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqbws\" (UniqueName: \"kubernetes.io/projected/20ddc125-fd2e-408a-9750-e817f2d9bf6c-kube-api-access-jqbws\") pod \"e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:03.394875 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.394739 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ddc125-fd2e-408a-9750-e817f2d9bf6c-credential-socket\") pod \"e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:03.394875 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.394774 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20ddc125-fd2e-408a-9750-e817f2d9bf6c-workload-certs\") pod \"e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:03.394875 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.394781 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20ddc125-fd2e-408a-9750-e817f2d9bf6c-istio-data\") pod \"e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:03.395241 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.395210 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20ddc125-fd2e-408a-9750-e817f2d9bf6c-istiod-ca-cert\") pod \"e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:03.395367 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.395286 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ddc125-fd2e-408a-9750-e817f2d9bf6c-workload-socket\") pod \"e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:03.397301 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.397274 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20ddc125-fd2e-408a-9750-e817f2d9bf6c-istio-podinfo\") pod \"e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:03.397382 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.397300 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20ddc125-fd2e-408a-9750-e817f2d9bf6c-istio-envoy\") pod \"e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:03.403579 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.403549 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20ddc125-fd2e-408a-9750-e817f2d9bf6c-istio-token\") pod \"e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:03.403696 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.403641 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqbws\" (UniqueName: \"kubernetes.io/projected/20ddc125-fd2e-408a-9750-e817f2d9bf6c-kube-api-access-jqbws\") pod \"e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:03.481817 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.481724 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:03.629757 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.629729 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25"] Jul 07 18:30:03.632174 ip-10-0-136-192 kubenswrapper[2583]: W0707 18:30:03.632149 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ddc125_fd2e_408a_9750_e817f2d9bf6c.slice/crio-1c268d5fc0f264e59383fdf6cee7264cf077e29d48002a1006abcc717ac7915e WatchSource:0}: Error finding container 1c268d5fc0f264e59383fdf6cee7264cf077e29d48002a1006abcc717ac7915e: Status 404 returned error can't find the container with id 1c268d5fc0f264e59383fdf6cee7264cf077e29d48002a1006abcc717ac7915e Jul 07 18:30:03.634493 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.634456 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:30:03.634608 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.634542 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:30:03.634608 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:03.634588 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:30:04.014531 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:04.014494 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" event={"ID":"20ddc125-fd2e-408a-9750-e817f2d9bf6c","Type":"ContainerStarted","Data":"e4662fa7d8520909483cdbb300d283b7855caaea635b5caae8f9b8d124abf5a3"} Jul 07 18:30:04.014697 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:04.014546 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" event={"ID":"20ddc125-fd2e-408a-9750-e817f2d9bf6c","Type":"ContainerStarted","Data":"1c268d5fc0f264e59383fdf6cee7264cf077e29d48002a1006abcc717ac7915e"} Jul 07 18:30:04.038678 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:04.038625 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" podStartSLOduration=1.038608065 podStartE2EDuration="1.038608065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:30:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:30:04.03727005 +0000 UTC m=+2153.119873750" watchObservedRunningTime="2026-07-07 18:30:04.038608065 +0000 UTC m=+2153.121211769" Jul 07 18:30:04.482985 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:04.482941 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:04.488052 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:04.488025 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:04.991503 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:04.991468 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-2b5160-58f958d857-rwj96" Jul 07 18:30:05.022635 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:05.022589 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:05.024772 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:05.024743 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:06.079925 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:06.079857 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:06.181165 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:06.181131 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:06.304171 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:06.304134 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln"] Jul 07 18:30:06.310008 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:06.308961 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln" Jul 07 18:30:06.313326 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:06.313292 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-200b05-serving-cert\"" Jul 07 18:30:06.326150 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:06.326123 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln"] Jul 07 18:30:06.366326 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:06.366229 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:06.425838 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:06.425787 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxd8j\" (UniqueName: \"kubernetes.io/projected/cd00ccaa-2e99-442d-83ff-9e776a01255c-kube-api-access-pxd8j\") pod \"maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln\" (UID: \"cd00ccaa-2e99-442d-83ff-9e776a01255c\") " pod="opendatahub/maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln" Jul 07 18:30:06.425838 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:06.425844 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd00ccaa-2e99-442d-83ff-9e776a01255c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln\" (UID: \"cd00ccaa-2e99-442d-83ff-9e776a01255c\") " pod="opendatahub/maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln" Jul 07 18:30:06.426079 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:06.425975 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd00ccaa-2e99-442d-83ff-9e776a01255c-maas-api-tls\") pod \"maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln\" (UID: \"cd00ccaa-2e99-442d-83ff-9e776a01255c\") " pod="opendatahub/maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln" Jul 07 18:30:06.465067 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:06.465032 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:06.526687 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:06.526644 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxd8j\" (UniqueName: \"kubernetes.io/projected/cd00ccaa-2e99-442d-83ff-9e776a01255c-kube-api-access-pxd8j\") pod \"maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln\" (UID: \"cd00ccaa-2e99-442d-83ff-9e776a01255c\") " pod="opendatahub/maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln" Jul 07 18:30:06.526687 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:06.526688 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd00ccaa-2e99-442d-83ff-9e776a01255c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln\" (UID: \"cd00ccaa-2e99-442d-83ff-9e776a01255c\") " pod="opendatahub/maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln" Jul 07 18:30:06.526893 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:06.526740 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd00ccaa-2e99-442d-83ff-9e776a01255c-maas-api-tls\") pod \"maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln\" (UID: \"cd00ccaa-2e99-442d-83ff-9e776a01255c\") " pod="opendatahub/maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln" Jul 07 18:30:06.527598 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:06.527527 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd00ccaa-2e99-442d-83ff-9e776a01255c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln\" (UID: \"cd00ccaa-2e99-442d-83ff-9e776a01255c\") " pod="opendatahub/maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln" Jul 07 18:30:06.529562 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:06.529524 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd00ccaa-2e99-442d-83ff-9e776a01255c-maas-api-tls\") pod \"maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln\" (UID: \"cd00ccaa-2e99-442d-83ff-9e776a01255c\") " pod="opendatahub/maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln" Jul 07 18:30:06.535753 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:06.535726 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxd8j\" (UniqueName: \"kubernetes.io/projected/cd00ccaa-2e99-442d-83ff-9e776a01255c-kube-api-access-pxd8j\") pod \"maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln\" (UID: \"cd00ccaa-2e99-442d-83ff-9e776a01255c\") " pod="opendatahub/maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln" Jul 07 18:30:06.625274 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:06.625172 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln" Jul 07 18:30:06.806166 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:06.802500 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln"] Jul 07 18:30:06.828376 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:06.828340 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:06.926434 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:06.926338 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:07.032883 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:07.032847 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln" event={"ID":"cd00ccaa-2e99-442d-83ff-9e776a01255c","Type":"ContainerStarted","Data":"6abb2499645207b853c53b2fd898c9b89c97bfcca213c3181426363873b13fbc"} Jul 07 18:30:08.039974 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:08.039939 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln" event={"ID":"cd00ccaa-2e99-442d-83ff-9e776a01255c","Type":"ContainerStarted","Data":"37bdc621b50f327ded4c9aa02d2017b5a0ccda944c1fae2bfcff73bae5655042"} Jul 07 18:30:08.040363 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:08.040011 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln" Jul 07 18:30:08.059541 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:08.059482 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln" podStartSLOduration=1.7221597100000001 podStartE2EDuration="2.059459148s" totalImagesPullingTime="337.299438ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 18:30:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:30:08.058415725 +0000 UTC m=+2157.141019430" watchObservedRunningTime="2026-07-07 18:30:08.059459148 +0000 UTC m=+2157.142062853" Jul 07 18:30:08.751994 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:08.751948 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:30:10.514059 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:10.514029 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:12.572568 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:12.572081 2583 patch_prober.go:28] interesting pod/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": read tcp 10.133.0.2:55524->10.133.0.47:15021: read: connection reset by peer" start-of-body= Jul 07 18:30:12.572568 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:12.572148 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" podUID="678edfb4-f4ee-41e4-af93-f2a741db1657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": read tcp 10.133.0.2:55524->10.133.0.47:15021: read: connection reset by peer" Jul 07 18:30:13.068983 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:13.068947 2583 generic.go:356] "Generic (PLEG): container finished" podID="678edfb4-f4ee-41e4-af93-f2a741db1657" containerID="0246789ae6357092cdc749c79c4d386739baba33ab3a6e810ee42d122b099dc6" exitCode=0 Jul 07 18:30:13.069171 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:13.069022 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" event={"ID":"678edfb4-f4ee-41e4-af93-f2a741db1657","Type":"ContainerDied","Data":"0246789ae6357092cdc749c79c4d386739baba33ab3a6e810ee42d122b099dc6"} Jul 07 18:30:13.069462 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:13.069448 2583 scope.go:117] "RemoveContainer" containerID="0246789ae6357092cdc749c79c4d386739baba33ab3a6e810ee42d122b099dc6" Jul 07 18:30:13.070572 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:13.070543 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:30:13.070631 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:13.070614 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:30:13.070667 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:13.070659 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:30:14.052099 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:14.052070 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln" Jul 07 18:30:14.077277 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:14.077231 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" event={"ID":"678edfb4-f4ee-41e4-af93-f2a741db1657","Type":"ContainerStarted","Data":"e7fa26065a17750024ce9e46306ed5a8568fb8b02dba45acc4497af375b45fcd"} Jul 07 18:30:14.259646 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:14.259606 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:30:14.260002 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:14.259967 2583 patch_prober.go:28] interesting pod/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 07 18:30:14.260102 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:14.260031 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" podUID="678edfb4-f4ee-41e4-af93-f2a741db1657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 07 18:30:15.259818 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:15.259766 2583 patch_prober.go:28] interesting pod/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 07 18:30:15.260231 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:15.259842 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" podUID="678edfb4-f4ee-41e4-af93-f2a741db1657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 07 18:30:15.724686 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:15.724653 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:16.259815 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:16.259781 2583 patch_prober.go:28] interesting pod/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 07 18:30:16.260029 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:16.259836 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" podUID="678edfb4-f4ee-41e4-af93-f2a741db1657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 07 18:30:17.259933 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:17.259882 2583 patch_prober.go:28] interesting pod/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 07 18:30:17.261082 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:17.259952 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" podUID="678edfb4-f4ee-41e4-af93-f2a741db1657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 07 18:30:18.106342 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:18.106310 2583 generic.go:356] "Generic (PLEG): container finished" podID="20ddc125-fd2e-408a-9750-e817f2d9bf6c" containerID="e4662fa7d8520909483cdbb300d283b7855caaea635b5caae8f9b8d124abf5a3" exitCode=0 Jul 07 18:30:18.106536 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:18.106381 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" event={"ID":"20ddc125-fd2e-408a-9750-e817f2d9bf6c","Type":"ContainerDied","Data":"e4662fa7d8520909483cdbb300d283b7855caaea635b5caae8f9b8d124abf5a3"} Jul 07 18:30:18.106853 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:18.106838 2583 scope.go:117] "RemoveContainer" containerID="e4662fa7d8520909483cdbb300d283b7855caaea635b5caae8f9b8d124abf5a3" Jul 07 18:30:18.107963 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:18.107928 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:30:18.108078 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:18.108018 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:30:18.108078 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:18.108065 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 07 18:30:18.263456 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:18.263415 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:30:18.263982 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:18.263928 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:30:18.264931 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:18.264871 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:30:19.114093 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:19.114052 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" event={"ID":"20ddc125-fd2e-408a-9750-e817f2d9bf6c","Type":"ContainerStarted","Data":"28eb0040ce7a4ecb98d75c5f91dccff8b2b7ed88138bc2c4255e249343ded98f"} Jul 07 18:30:19.404567 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:19.404449 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:19.482213 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:19.482169 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:19.483499 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:19.483471 2583 patch_prober.go:28] interesting pod/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 07 18:30:19.483563 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:19.483528 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" podUID="20ddc125-fd2e-408a-9750-e817f2d9bf6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 07 18:30:19.504866 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:19.504710 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:19.936332 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:19.936293 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:30:20.023018 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:20.022975 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:20.482394 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:20.482351 2583 patch_prober.go:28] interesting pod/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 07 18:30:20.482814 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:20.482422 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" podUID="20ddc125-fd2e-408a-9750-e817f2d9bf6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 07 18:30:21.482553 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:21.482511 2583 patch_prober.go:28] interesting pod/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 07 18:30:21.482990 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:21.482568 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" podUID="20ddc125-fd2e-408a-9750-e817f2d9bf6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 07 18:30:22.483061 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:22.483018 2583 patch_prober.go:28] interesting pod/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 07 18:30:22.483484 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:22.483089 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" podUID="20ddc125-fd2e-408a-9750-e817f2d9bf6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 07 18:30:23.488327 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:23.488294 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:23.489312 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:23.489291 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:30:27.548273 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:27.548218 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:27.648526 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:27.648455 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:29.778835 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:29.778803 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:30:33.164891 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:33.164855 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:30:45.670151 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:30:45.670115 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:31:02.061289 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:31:02.061250 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:31:17.853958 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:31:17.853923 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:31:35.044604 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:31:35.044512 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:31:52.942453 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:31:52.942416 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:31:53.415857 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:31:53.415821 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:32:09.743875 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:32:09.743838 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:32:23.697517 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:32:23.697478 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:32:24.610608 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:32:24.610572 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:32:26.838811 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:32:26.838771 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:32:37.012691 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:32:37.012633 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:32:41.443599 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:32:41.443561 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:32:52.818435 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:32:52.818390 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:32:54.614519 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:32:54.614481 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:32:58.073458 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:32:58.073424 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:33:08.544618 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:33:08.544541 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:33:28.920470 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:33:28.920429 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:33:29.009136 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:33:29.009099 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:33:35.352377 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:33:35.352336 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:33:52.012157 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:33:52.012121 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:33:52.115275 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:33:52.115224 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:33:52.943533 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:33:52.943493 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:33:59.592340 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:33:59.592302 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:34:00.513696 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:34:00.513662 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:34:09.943524 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:34:09.943486 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:34:11.524777 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:34:11.524748 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wqfm4_997df85c-7632-47b5-a8a6-1c37f3346251/console-operator/2.log" Jul 07 18:34:11.527156 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:34:11.527133 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wqfm4_997df85c-7632-47b5-a8a6-1c37f3346251/console-operator/2.log" Jul 07 18:34:11.531531 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:34:11.531507 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zjzpw_5c2d819c-e495-47ec-a1e7-4b2582a195bd/ovn-acl-logging/0.log" Jul 07 18:34:11.533926 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:34:11.533893 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zjzpw_5c2d819c-e495-47ec-a1e7-4b2582a195bd/ovn-acl-logging/0.log" Jul 07 18:34:15.818122 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:34:15.818084 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:34:20.739324 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:34:20.739285 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:34:34.396347 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:34:34.396310 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:34:37.893135 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:34:37.893102 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:34:40.497122 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:34:40.497088 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:35:05.752442 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:05.752401 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:35:19.552280 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:19.552239 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:35:20.269606 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:20.269557 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:35:20.370172 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:20.370135 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:35:27.291812 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:27.291768 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:35:28.195643 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:28.195608 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:35:28.320721 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:28.320690 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:35:28.466197 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:28.466083 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:35:29.478212 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:29.478178 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:35:29.954844 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:29.954813 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:35:30.087522 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:30.087477 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:35:30.201480 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:30.201446 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:35:30.357282 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:30.357236 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-2b5160-58f958d857-rwj96"] Jul 07 18:35:30.357519 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:30.357489 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-2b5160-58f958d857-rwj96" podUID="d12baac8-ff09-4c0c-879a-0914c5dcaef3" containerName="maas-api" containerID="cri-o://70f5b11893c1fb9aa8395e51098301dea3d3ec35f77f8a3b9455e9f0494596f5" gracePeriod=30 Jul 07 18:35:30.387154 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:30.387115 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:35:30.616553 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:30.616467 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-2b5160-58f958d857-rwj96" Jul 07 18:35:30.687770 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:30.687732 2583 generic.go:356] "Generic (PLEG): container finished" podID="d12baac8-ff09-4c0c-879a-0914c5dcaef3" containerID="70f5b11893c1fb9aa8395e51098301dea3d3ec35f77f8a3b9455e9f0494596f5" exitCode=0 Jul 07 18:35:30.687975 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:30.687783 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-2b5160-58f958d857-rwj96" Jul 07 18:35:30.687975 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:30.687797 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-2b5160-58f958d857-rwj96" event={"ID":"d12baac8-ff09-4c0c-879a-0914c5dcaef3","Type":"ContainerDied","Data":"70f5b11893c1fb9aa8395e51098301dea3d3ec35f77f8a3b9455e9f0494596f5"} Jul 07 18:35:30.687975 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:30.687841 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-2b5160-58f958d857-rwj96" event={"ID":"d12baac8-ff09-4c0c-879a-0914c5dcaef3","Type":"ContainerDied","Data":"eb4cd0b34f123cff0117984bcde0aa3677acdd91f1689bb0e8b6869b3e21857e"} Jul 07 18:35:30.687975 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:30.687861 2583 scope.go:117] "RemoveContainer" containerID="70f5b11893c1fb9aa8395e51098301dea3d3ec35f77f8a3b9455e9f0494596f5" Jul 07 18:35:30.697096 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:30.697075 2583 scope.go:117] "RemoveContainer" containerID="70f5b11893c1fb9aa8395e51098301dea3d3ec35f77f8a3b9455e9f0494596f5" Jul 07 18:35:30.697375 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:35:30.697357 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70f5b11893c1fb9aa8395e51098301dea3d3ec35f77f8a3b9455e9f0494596f5\": container with ID starting with 70f5b11893c1fb9aa8395e51098301dea3d3ec35f77f8a3b9455e9f0494596f5 not found: ID does not exist" containerID="70f5b11893c1fb9aa8395e51098301dea3d3ec35f77f8a3b9455e9f0494596f5" Jul 07 18:35:30.697435 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:30.697383 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70f5b11893c1fb9aa8395e51098301dea3d3ec35f77f8a3b9455e9f0494596f5"} err="failed to get container status \"70f5b11893c1fb9aa8395e51098301dea3d3ec35f77f8a3b9455e9f0494596f5\": rpc error: code = NotFound desc = could not find container \"70f5b11893c1fb9aa8395e51098301dea3d3ec35f77f8a3b9455e9f0494596f5\": container with ID starting with 70f5b11893c1fb9aa8395e51098301dea3d3ec35f77f8a3b9455e9f0494596f5 not found: ID does not exist" Jul 07 18:35:30.719433 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:30.719403 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jnmr\" (UniqueName: \"kubernetes.io/projected/d12baac8-ff09-4c0c-879a-0914c5dcaef3-kube-api-access-7jnmr\") pod \"d12baac8-ff09-4c0c-879a-0914c5dcaef3\" (UID: \"d12baac8-ff09-4c0c-879a-0914c5dcaef3\") " Jul 07 18:35:30.719605 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:30.719558 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d12baac8-ff09-4c0c-879a-0914c5dcaef3-openshift-service-ca\") pod \"d12baac8-ff09-4c0c-879a-0914c5dcaef3\" (UID: \"d12baac8-ff09-4c0c-879a-0914c5dcaef3\") " Jul 07 18:35:30.719605 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:30.719577 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d12baac8-ff09-4c0c-879a-0914c5dcaef3-maas-api-tls\") pod \"d12baac8-ff09-4c0c-879a-0914c5dcaef3\" (UID: \"d12baac8-ff09-4c0c-879a-0914c5dcaef3\") " Jul 07 18:35:30.719917 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:30.719880 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12baac8-ff09-4c0c-879a-0914c5dcaef3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d12baac8-ff09-4c0c-879a-0914c5dcaef3" (UID: "d12baac8-ff09-4c0c-879a-0914c5dcaef3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:35:30.721834 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:30.721808 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12baac8-ff09-4c0c-879a-0914c5dcaef3-kube-api-access-7jnmr" (OuterVolumeSpecName: "kube-api-access-7jnmr") pod "d12baac8-ff09-4c0c-879a-0914c5dcaef3" (UID: "d12baac8-ff09-4c0c-879a-0914c5dcaef3"). InnerVolumeSpecName "kube-api-access-7jnmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:35:30.722023 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:30.722005 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d12baac8-ff09-4c0c-879a-0914c5dcaef3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d12baac8-ff09-4c0c-879a-0914c5dcaef3" (UID: "d12baac8-ff09-4c0c-879a-0914c5dcaef3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:35:30.820819 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:30.820765 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d12baac8-ff09-4c0c-879a-0914c5dcaef3-openshift-service-ca\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:35:30.820819 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:30.820816 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d12baac8-ff09-4c0c-879a-0914c5dcaef3-maas-api-tls\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:35:30.820819 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:30.820827 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jnmr\" (UniqueName: \"kubernetes.io/projected/d12baac8-ff09-4c0c-879a-0914c5dcaef3-kube-api-access-7jnmr\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:35:31.020291 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:31.020249 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-2b5160-58f958d857-rwj96"] Jul 07 18:35:31.026730 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:31.026697 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-2b5160-58f958d857-rwj96"] Jul 07 18:35:31.429604 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:31.429565 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d12baac8-ff09-4c0c-879a-0914c5dcaef3" path="/var/lib/kubelet/pods/d12baac8-ff09-4c0c-879a-0914c5dcaef3/volumes" Jul 07 18:35:32.776160 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:32.776125 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:35:32.867797 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:32.867762 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:35:34.263311 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:34.263272 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:35:35.030654 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:35:35.030621 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:36:14.007251 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:14.007211 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:36:14.341618 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:14.341579 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s"] Jul 07 18:36:14.341837 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:14.341798 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" podUID="678edfb4-f4ee-41e4-af93-f2a741db1657" containerName="istio-proxy" containerID="cri-o://e7fa26065a17750024ce9e46306ed5a8568fb8b02dba45acc4497af375b45fcd" gracePeriod=30 Jul 07 18:36:14.417098 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:14.417032 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:36:14.650721 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:14.650619 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:36:14.941594 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:14.941497 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:36:15.372331 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:15.372294 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:36:15.769312 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:15.769213 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:36:15.933192 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:15.933147 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:36:16.155706 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:16.155668 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln"] Jul 07 18:36:16.155960 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:16.155932 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln" podUID="cd00ccaa-2e99-442d-83ff-9e776a01255c" containerName="maas-api" containerID="cri-o://37bdc621b50f327ded4c9aa02d2017b5a0ccda944c1fae2bfcff73bae5655042" gracePeriod=30 Jul 07 18:36:16.248274 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:16.248234 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:36:16.418756 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:16.418683 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln" Jul 07 18:36:16.557971 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:16.557933 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxd8j\" (UniqueName: \"kubernetes.io/projected/cd00ccaa-2e99-442d-83ff-9e776a01255c-kube-api-access-pxd8j\") pod \"cd00ccaa-2e99-442d-83ff-9e776a01255c\" (UID: \"cd00ccaa-2e99-442d-83ff-9e776a01255c\") " Jul 07 18:36:16.558186 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:16.557992 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd00ccaa-2e99-442d-83ff-9e776a01255c-maas-api-tls\") pod \"cd00ccaa-2e99-442d-83ff-9e776a01255c\" (UID: \"cd00ccaa-2e99-442d-83ff-9e776a01255c\") " Jul 07 18:36:16.558186 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:16.558100 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd00ccaa-2e99-442d-83ff-9e776a01255c-openshift-service-ca\") pod \"cd00ccaa-2e99-442d-83ff-9e776a01255c\" (UID: \"cd00ccaa-2e99-442d-83ff-9e776a01255c\") " Jul 07 18:36:16.558450 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:16.558425 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd00ccaa-2e99-442d-83ff-9e776a01255c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cd00ccaa-2e99-442d-83ff-9e776a01255c" (UID: "cd00ccaa-2e99-442d-83ff-9e776a01255c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:36:16.560186 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:16.560164 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd00ccaa-2e99-442d-83ff-9e776a01255c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cd00ccaa-2e99-442d-83ff-9e776a01255c" (UID: "cd00ccaa-2e99-442d-83ff-9e776a01255c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 18:36:16.560298 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:16.560280 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd00ccaa-2e99-442d-83ff-9e776a01255c-kube-api-access-pxd8j" (OuterVolumeSpecName: "kube-api-access-pxd8j") pod "cd00ccaa-2e99-442d-83ff-9e776a01255c" (UID: "cd00ccaa-2e99-442d-83ff-9e776a01255c"). InnerVolumeSpecName "kube-api-access-pxd8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:36:16.659785 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:16.659745 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd00ccaa-2e99-442d-83ff-9e776a01255c-maas-api-tls\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:36:16.659785 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:16.659778 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd00ccaa-2e99-442d-83ff-9e776a01255c-openshift-service-ca\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:36:16.659785 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:16.659790 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxd8j\" (UniqueName: \"kubernetes.io/projected/cd00ccaa-2e99-442d-83ff-9e776a01255c-kube-api-access-pxd8j\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:36:16.920979 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:16.920943 2583 generic.go:356] "Generic (PLEG): container finished" podID="cd00ccaa-2e99-442d-83ff-9e776a01255c" containerID="37bdc621b50f327ded4c9aa02d2017b5a0ccda944c1fae2bfcff73bae5655042" exitCode=0 Jul 07 18:36:16.921204 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:16.920989 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln" Jul 07 18:36:16.921204 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:16.921031 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln" event={"ID":"cd00ccaa-2e99-442d-83ff-9e776a01255c","Type":"ContainerDied","Data":"37bdc621b50f327ded4c9aa02d2017b5a0ccda944c1fae2bfcff73bae5655042"} Jul 07 18:36:16.921204 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:16.921089 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln" event={"ID":"cd00ccaa-2e99-442d-83ff-9e776a01255c","Type":"ContainerDied","Data":"6abb2499645207b853c53b2fd898c9b89c97bfcca213c3181426363873b13fbc"} Jul 07 18:36:16.921204 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:16.921117 2583 scope.go:117] "RemoveContainer" containerID="37bdc621b50f327ded4c9aa02d2017b5a0ccda944c1fae2bfcff73bae5655042" Jul 07 18:36:16.931318 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:16.931299 2583 scope.go:117] "RemoveContainer" containerID="37bdc621b50f327ded4c9aa02d2017b5a0ccda944c1fae2bfcff73bae5655042" Jul 07 18:36:16.931656 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:36:16.931628 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37bdc621b50f327ded4c9aa02d2017b5a0ccda944c1fae2bfcff73bae5655042\": container with ID starting with 37bdc621b50f327ded4c9aa02d2017b5a0ccda944c1fae2bfcff73bae5655042 not found: ID does not exist" containerID="37bdc621b50f327ded4c9aa02d2017b5a0ccda944c1fae2bfcff73bae5655042" Jul 07 18:36:16.931756 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:16.931665 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37bdc621b50f327ded4c9aa02d2017b5a0ccda944c1fae2bfcff73bae5655042"} err="failed to get container status \"37bdc621b50f327ded4c9aa02d2017b5a0ccda944c1fae2bfcff73bae5655042\": rpc error: code = NotFound desc = could not find container \"37bdc621b50f327ded4c9aa02d2017b5a0ccda944c1fae2bfcff73bae5655042\": container with ID starting with 37bdc621b50f327ded4c9aa02d2017b5a0ccda944c1fae2bfcff73bae5655042 not found: ID does not exist" Jul 07 18:36:16.951941 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:16.951895 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln"] Jul 07 18:36:16.966764 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:16.966735 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-200b05-5c57489fcc-jtcln"] Jul 07 18:36:17.429231 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:17.429200 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd00ccaa-2e99-442d-83ff-9e776a01255c" path="/var/lib/kubelet/pods/cd00ccaa-2e99-442d-83ff-9e776a01255c/volumes" Jul 07 18:36:18.266494 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:18.266455 2583 patch_prober.go:28] interesting pod/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 18:36:18.266682 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:18.266509 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" podUID="678edfb4-f4ee-41e4-af93-f2a741db1657" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 18:36:18.539760 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:18.539656 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jfmtn"] Jul 07 18:36:19.602191 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.602119 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:36:19.791648 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.791604 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/678edfb4-f4ee-41e4-af93-f2a741db1657-istio-data\") pod \"678edfb4-f4ee-41e4-af93-f2a741db1657\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " Jul 07 18:36:19.791648 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.791649 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/678edfb4-f4ee-41e4-af93-f2a741db1657-istio-podinfo\") pod \"678edfb4-f4ee-41e4-af93-f2a741db1657\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " Jul 07 18:36:19.791954 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.791682 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/678edfb4-f4ee-41e4-af93-f2a741db1657-workload-socket\") pod \"678edfb4-f4ee-41e4-af93-f2a741db1657\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " Jul 07 18:36:19.791954 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.791740 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/678edfb4-f4ee-41e4-af93-f2a741db1657-istio-envoy\") pod \"678edfb4-f4ee-41e4-af93-f2a741db1657\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " Jul 07 18:36:19.791954 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.791770 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dbqg\" (UniqueName: \"kubernetes.io/projected/678edfb4-f4ee-41e4-af93-f2a741db1657-kube-api-access-2dbqg\") pod \"678edfb4-f4ee-41e4-af93-f2a741db1657\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " Jul 07 18:36:19.791954 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.791803 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/678edfb4-f4ee-41e4-af93-f2a741db1657-workload-certs\") pod \"678edfb4-f4ee-41e4-af93-f2a741db1657\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " Jul 07 18:36:19.791954 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.791831 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/678edfb4-f4ee-41e4-af93-f2a741db1657-istio-token\") pod \"678edfb4-f4ee-41e4-af93-f2a741db1657\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " Jul 07 18:36:19.792224 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.791975 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/678edfb4-f4ee-41e4-af93-f2a741db1657-credential-socket\") pod \"678edfb4-f4ee-41e4-af93-f2a741db1657\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " Jul 07 18:36:19.792224 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.792000 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/678edfb4-f4ee-41e4-af93-f2a741db1657-istiod-ca-cert\") pod \"678edfb4-f4ee-41e4-af93-f2a741db1657\" (UID: \"678edfb4-f4ee-41e4-af93-f2a741db1657\") " Jul 07 18:36:19.792224 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.792119 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/678edfb4-f4ee-41e4-af93-f2a741db1657-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "678edfb4-f4ee-41e4-af93-f2a741db1657" (UID: "678edfb4-f4ee-41e4-af93-f2a741db1657"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:36:19.792224 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.792139 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/678edfb4-f4ee-41e4-af93-f2a741db1657-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "678edfb4-f4ee-41e4-af93-f2a741db1657" (UID: "678edfb4-f4ee-41e4-af93-f2a741db1657"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:36:19.792428 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.792402 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/678edfb4-f4ee-41e4-af93-f2a741db1657-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "678edfb4-f4ee-41e4-af93-f2a741db1657" (UID: "678edfb4-f4ee-41e4-af93-f2a741db1657"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:36:19.792530 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.792448 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/678edfb4-f4ee-41e4-af93-f2a741db1657-workload-certs\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:36:19.792530 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.792471 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/678edfb4-f4ee-41e4-af93-f2a741db1657-workload-socket\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:36:19.792530 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.792531 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/678edfb4-f4ee-41e4-af93-f2a741db1657-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "678edfb4-f4ee-41e4-af93-f2a741db1657" (UID: "678edfb4-f4ee-41e4-af93-f2a741db1657"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:36:19.792808 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.792715 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/678edfb4-f4ee-41e4-af93-f2a741db1657-istio-data" (OuterVolumeSpecName: "istio-data") pod "678edfb4-f4ee-41e4-af93-f2a741db1657" (UID: "678edfb4-f4ee-41e4-af93-f2a741db1657"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:36:19.794162 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.794138 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/678edfb4-f4ee-41e4-af93-f2a741db1657-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "678edfb4-f4ee-41e4-af93-f2a741db1657" (UID: "678edfb4-f4ee-41e4-af93-f2a741db1657"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 18:36:19.794421 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.794402 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/678edfb4-f4ee-41e4-af93-f2a741db1657-kube-api-access-2dbqg" (OuterVolumeSpecName: "kube-api-access-2dbqg") pod "678edfb4-f4ee-41e4-af93-f2a741db1657" (UID: "678edfb4-f4ee-41e4-af93-f2a741db1657"). InnerVolumeSpecName "kube-api-access-2dbqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:36:19.794688 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.794670 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/678edfb4-f4ee-41e4-af93-f2a741db1657-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "678edfb4-f4ee-41e4-af93-f2a741db1657" (UID: "678edfb4-f4ee-41e4-af93-f2a741db1657"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:36:19.794731 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.794682 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/678edfb4-f4ee-41e4-af93-f2a741db1657-istio-token" (OuterVolumeSpecName: "istio-token") pod "678edfb4-f4ee-41e4-af93-f2a741db1657" (UID: "678edfb4-f4ee-41e4-af93-f2a741db1657"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:36:19.893718 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.893619 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/678edfb4-f4ee-41e4-af93-f2a741db1657-credential-socket\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:36:19.893718 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.893654 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/678edfb4-f4ee-41e4-af93-f2a741db1657-istiod-ca-cert\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:36:19.893718 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.893666 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/678edfb4-f4ee-41e4-af93-f2a741db1657-istio-data\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:36:19.893718 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.893675 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/678edfb4-f4ee-41e4-af93-f2a741db1657-istio-podinfo\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:36:19.893718 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.893685 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/678edfb4-f4ee-41e4-af93-f2a741db1657-istio-envoy\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:36:19.893718 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.893695 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dbqg\" (UniqueName: \"kubernetes.io/projected/678edfb4-f4ee-41e4-af93-f2a741db1657-kube-api-access-2dbqg\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:36:19.893718 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.893703 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/678edfb4-f4ee-41e4-af93-f2a741db1657-istio-token\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:36:19.940541 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.940504 2583 generic.go:356] "Generic (PLEG): container finished" podID="678edfb4-f4ee-41e4-af93-f2a741db1657" containerID="e7fa26065a17750024ce9e46306ed5a8568fb8b02dba45acc4497af375b45fcd" exitCode=0 Jul 07 18:36:19.940718 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.940551 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" Jul 07 18:36:19.940718 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.940589 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" event={"ID":"678edfb4-f4ee-41e4-af93-f2a741db1657","Type":"ContainerDied","Data":"e7fa26065a17750024ce9e46306ed5a8568fb8b02dba45acc4497af375b45fcd"} Jul 07 18:36:19.940718 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.940650 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s" event={"ID":"678edfb4-f4ee-41e4-af93-f2a741db1657","Type":"ContainerDied","Data":"72224ed065df7352a5ab61d879819a4e2b9cf26482b83135b1c0845ea9b40847"} Jul 07 18:36:19.940718 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.940673 2583 scope.go:117] "RemoveContainer" containerID="e7fa26065a17750024ce9e46306ed5a8568fb8b02dba45acc4497af375b45fcd" Jul 07 18:36:19.950353 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.950332 2583 scope.go:117] "RemoveContainer" containerID="0246789ae6357092cdc749c79c4d386739baba33ab3a6e810ee42d122b099dc6" Jul 07 18:36:19.959263 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.959242 2583 scope.go:117] "RemoveContainer" containerID="e7fa26065a17750024ce9e46306ed5a8568fb8b02dba45acc4497af375b45fcd" Jul 07 18:36:19.959574 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:36:19.959551 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7fa26065a17750024ce9e46306ed5a8568fb8b02dba45acc4497af375b45fcd\": container with ID starting with e7fa26065a17750024ce9e46306ed5a8568fb8b02dba45acc4497af375b45fcd not found: ID does not exist" containerID="e7fa26065a17750024ce9e46306ed5a8568fb8b02dba45acc4497af375b45fcd" Jul 07 18:36:19.959616 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.959585 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7fa26065a17750024ce9e46306ed5a8568fb8b02dba45acc4497af375b45fcd"} err="failed to get container status \"e7fa26065a17750024ce9e46306ed5a8568fb8b02dba45acc4497af375b45fcd\": rpc error: code = NotFound desc = could not find container \"e7fa26065a17750024ce9e46306ed5a8568fb8b02dba45acc4497af375b45fcd\": container with ID starting with e7fa26065a17750024ce9e46306ed5a8568fb8b02dba45acc4497af375b45fcd not found: ID does not exist" Jul 07 18:36:19.959616 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.959605 2583 scope.go:117] "RemoveContainer" containerID="0246789ae6357092cdc749c79c4d386739baba33ab3a6e810ee42d122b099dc6" Jul 07 18:36:19.959885 ip-10-0-136-192 kubenswrapper[2583]: E0707 18:36:19.959867 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0246789ae6357092cdc749c79c4d386739baba33ab3a6e810ee42d122b099dc6\": container with ID starting with 0246789ae6357092cdc749c79c4d386739baba33ab3a6e810ee42d122b099dc6 not found: ID does not exist" containerID="0246789ae6357092cdc749c79c4d386739baba33ab3a6e810ee42d122b099dc6" Jul 07 18:36:19.959957 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.959890 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0246789ae6357092cdc749c79c4d386739baba33ab3a6e810ee42d122b099dc6"} err="failed to get container status \"0246789ae6357092cdc749c79c4d386739baba33ab3a6e810ee42d122b099dc6\": rpc error: code = NotFound desc = could not find container \"0246789ae6357092cdc749c79c4d386739baba33ab3a6e810ee42d122b099dc6\": container with ID starting with 0246789ae6357092cdc749c79c4d386739baba33ab3a6e810ee42d122b099dc6 not found: ID does not exist" Jul 07 18:36:19.965013 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.964983 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s"] Jul 07 18:36:19.970366 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:19.970331 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-2b5160-openshift-default-6c58ff44d7-gth4s"] Jul 07 18:36:21.435510 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:21.431509 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="678edfb4-f4ee-41e4-af93-f2a741db1657" path="/var/lib/kubelet/pods/678edfb4-f4ee-41e4-af93-f2a741db1657/volumes" Jul 07 18:36:22.454896 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:22.454846 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:36:22.557117 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:22.557081 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:36:22.785171 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:22.785086 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25"] Jul 07 18:36:22.785358 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:22.785333 2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" podUID="20ddc125-fd2e-408a-9750-e817f2d9bf6c" containerName="istio-proxy" containerID="cri-o://28eb0040ce7a4ecb98d75c5f91dccff8b2b7ed88138bc2c4255e249343ded98f" gracePeriod=30 Jul 07 18:36:22.809082 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:22.809051 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:36:23.098584 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:23.098552 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:36:23.491685 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:23.491601 2583 patch_prober.go:28] interesting pod/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 18:36:23.492078 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:23.491677 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" podUID="20ddc125-fd2e-408a-9750-e817f2d9bf6c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 18:36:27.982847 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:27.982804 2583 generic.go:356] "Generic (PLEG): container finished" podID="20ddc125-fd2e-408a-9750-e817f2d9bf6c" containerID="28eb0040ce7a4ecb98d75c5f91dccff8b2b7ed88138bc2c4255e249343ded98f" exitCode=0 Jul 07 18:36:27.983316 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:27.982847 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" event={"ID":"20ddc125-fd2e-408a-9750-e817f2d9bf6c","Type":"ContainerDied","Data":"28eb0040ce7a4ecb98d75c5f91dccff8b2b7ed88138bc2c4255e249343ded98f"} Jul 07 18:36:27.983316 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:27.982949 2583 scope.go:117] "RemoveContainer" containerID="e4662fa7d8520909483cdbb300d283b7855caaea635b5caae8f9b8d124abf5a3" Jul 07 18:36:28.046796 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:28.046764 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:36:28.065287 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:28.065201 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20ddc125-fd2e-408a-9750-e817f2d9bf6c-istio-envoy\") pod \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " Jul 07 18:36:28.065287 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:28.065245 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqbws\" (UniqueName: \"kubernetes.io/projected/20ddc125-fd2e-408a-9750-e817f2d9bf6c-kube-api-access-jqbws\") pod \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " Jul 07 18:36:28.065287 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:28.065285 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ddc125-fd2e-408a-9750-e817f2d9bf6c-workload-socket\") pod \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " Jul 07 18:36:28.065539 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:28.065339 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20ddc125-fd2e-408a-9750-e817f2d9bf6c-istio-token\") pod \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " Jul 07 18:36:28.065539 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:28.065366 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20ddc125-fd2e-408a-9750-e817f2d9bf6c-istiod-ca-cert\") pod \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " Jul 07 18:36:28.065539 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:28.065396 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20ddc125-fd2e-408a-9750-e817f2d9bf6c-workload-certs\") pod \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " Jul 07 18:36:28.065539 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:28.065526 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20ddc125-fd2e-408a-9750-e817f2d9bf6c-istio-podinfo\") pod \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " Jul 07 18:36:28.065757 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:28.065602 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ddc125-fd2e-408a-9750-e817f2d9bf6c-credential-socket\") pod \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " Jul 07 18:36:28.065757 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:28.065675 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20ddc125-fd2e-408a-9750-e817f2d9bf6c-istio-data\") pod \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\" (UID: \"20ddc125-fd2e-408a-9750-e817f2d9bf6c\") " Jul 07 18:36:28.065757 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:28.065694 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ddc125-fd2e-408a-9750-e817f2d9bf6c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "20ddc125-fd2e-408a-9750-e817f2d9bf6c" (UID: "20ddc125-fd2e-408a-9750-e817f2d9bf6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:36:28.065896 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:28.065816 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20ddc125-fd2e-408a-9750-e817f2d9bf6c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "20ddc125-fd2e-408a-9750-e817f2d9bf6c" (UID: "20ddc125-fd2e-408a-9750-e817f2d9bf6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 18:36:28.066186 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:28.065972 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ddc125-fd2e-408a-9750-e817f2d9bf6c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "20ddc125-fd2e-408a-9750-e817f2d9bf6c" (UID: "20ddc125-fd2e-408a-9750-e817f2d9bf6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:36:28.066186 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:28.066155 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ddc125-fd2e-408a-9750-e817f2d9bf6c-workload-socket\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:36:28.066186 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:28.066176 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20ddc125-fd2e-408a-9750-e817f2d9bf6c-istiod-ca-cert\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:36:28.066186 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:28.066189 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ddc125-fd2e-408a-9750-e817f2d9bf6c-credential-socket\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:36:28.066441 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:28.066324 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ddc125-fd2e-408a-9750-e817f2d9bf6c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "20ddc125-fd2e-408a-9750-e817f2d9bf6c" (UID: "20ddc125-fd2e-408a-9750-e817f2d9bf6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:36:28.066527 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:28.066508 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ddc125-fd2e-408a-9750-e817f2d9bf6c-istio-data" (OuterVolumeSpecName: "istio-data") pod "20ddc125-fd2e-408a-9750-e817f2d9bf6c" (UID: "20ddc125-fd2e-408a-9750-e817f2d9bf6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:36:28.067745 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:28.067721 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ddc125-fd2e-408a-9750-e817f2d9bf6c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "20ddc125-fd2e-408a-9750-e817f2d9bf6c" (UID: "20ddc125-fd2e-408a-9750-e817f2d9bf6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 18:36:28.067745 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:28.067739 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ddc125-fd2e-408a-9750-e817f2d9bf6c-istio-token" (OuterVolumeSpecName: "istio-token") pod "20ddc125-fd2e-408a-9750-e817f2d9bf6c" (UID: "20ddc125-fd2e-408a-9750-e817f2d9bf6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:36:28.068244 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:28.068225 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ddc125-fd2e-408a-9750-e817f2d9bf6c-kube-api-access-jqbws" (OuterVolumeSpecName: "kube-api-access-jqbws") pod "20ddc125-fd2e-408a-9750-e817f2d9bf6c" (UID: "20ddc125-fd2e-408a-9750-e817f2d9bf6c"). InnerVolumeSpecName "kube-api-access-jqbws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 18:36:28.068244 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:28.068226 2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20ddc125-fd2e-408a-9750-e817f2d9bf6c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "20ddc125-fd2e-408a-9750-e817f2d9bf6c" (UID: "20ddc125-fd2e-408a-9750-e817f2d9bf6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 18:36:28.167413 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:28.167381 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20ddc125-fd2e-408a-9750-e817f2d9bf6c-workload-certs\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:36:28.167413 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:28.167409 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20ddc125-fd2e-408a-9750-e817f2d9bf6c-istio-podinfo\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:36:28.167413 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:28.167418 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20ddc125-fd2e-408a-9750-e817f2d9bf6c-istio-data\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:36:28.167672 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:28.167428 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20ddc125-fd2e-408a-9750-e817f2d9bf6c-istio-envoy\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:36:28.167672 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:28.167438 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqbws\" (UniqueName: \"kubernetes.io/projected/20ddc125-fd2e-408a-9750-e817f2d9bf6c-kube-api-access-jqbws\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:36:28.167672 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:28.167447 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20ddc125-fd2e-408a-9750-e817f2d9bf6c-istio-token\") on node \"ip-10-0-136-192.ec2.internal\" DevicePath \"\"" Jul 07 18:36:28.989430 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:28.989401 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" Jul 07 18:36:28.989430 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:28.989408 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25" event={"ID":"20ddc125-fd2e-408a-9750-e817f2d9bf6c","Type":"ContainerDied","Data":"1c268d5fc0f264e59383fdf6cee7264cf077e29d48002a1006abcc717ac7915e"} Jul 07 18:36:28.989948 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:28.989470 2583 scope.go:117] "RemoveContainer" containerID="28eb0040ce7a4ecb98d75c5f91dccff8b2b7ed88138bc2c4255e249343ded98f" Jul 07 18:36:29.031190 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:29.031156 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25"] Jul 07 18:36:29.040571 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:29.040535 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-200b05-openshift-default-57cf859bd7-5cb25"] Jul 07 18:36:29.429533 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:29.429496 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20ddc125-fd2e-408a-9750-e817f2d9bf6c" path="/var/lib/kubelet/pods/20ddc125-fd2e-408a-9750-e817f2d9bf6c/volumes" Jul 07 18:36:30.837430 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:30.837399 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76cd65fb98-6cfbr_1bb166ca-6b5b-4ed8-af7c-97d60eed5e94/copy-manifests/0.log" Jul 07 18:36:30.864139 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:30.864110 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76cd65fb98-6cfbr_1bb166ca-6b5b-4ed8-af7c-97d60eed5e94/manager/0.log" Jul 07 18:36:33.827086 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:33.827054 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-jfmtn_4f756f4c-3846-4888-bc28-70c2dc1f9bfb/limitador/0.log" Jul 07 18:36:34.401336 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:34.401302 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4r59rt_6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2/istio-proxy/0.log" Jul 07 18:36:34.535151 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:34.535116 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-j4f4p_27c52354-e3bb-47b5-94ca-6ec736c41d3f/discovery/0.log" Jul 07 18:36:35.450918 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:35.450869 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d876b7659-t4r6h_4d9f9071-7f2b-4944-88fc-130da6a92ba5/router/0.log" Jul 07 18:36:40.891396 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:40.891362 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:36:41.154265 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:41.154179 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" limit=1024 Jul 07 18:36:44.483617 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:44.483591 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xqq9h_3a1d71c1-6f70-441c-87bb-1f997df1fd5d/global-pull-secret-syncer/0.log" Jul 07 18:36:44.604173 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:44.604139 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6zwkl_f78b3750-ae77-4cf9-af3e-4ec59bc87dde/konnectivity-agent/0.log" Jul 07 18:36:44.930849 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:44.930822 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-192.ec2.internal_cf831b3419f5fd8b4190fd80bde7e0ae/haproxy/0.log" Jul 07 18:36:49.798795 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:49.798764 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-jfmtn_4f756f4c-3846-4888-bc28-70c2dc1f9bfb/limitador/0.log" Jul 07 18:36:51.700840 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:51.700806 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b776ec07-9a59-4ed8-9f3a-7880c377d46a/alertmanager/0.log" Jul 07 18:36:51.777564 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:51.777529 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b776ec07-9a59-4ed8-9f3a-7880c377d46a/config-reloader/0.log" Jul 07 18:36:51.855654 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:51.855624 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b776ec07-9a59-4ed8-9f3a-7880c377d46a/kube-rbac-proxy-web/0.log" Jul 07 18:36:51.909897 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:51.909854 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b776ec07-9a59-4ed8-9f3a-7880c377d46a/kube-rbac-proxy/0.log" Jul 07 18:36:51.968537 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:51.968419 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b776ec07-9a59-4ed8-9f3a-7880c377d46a/kube-rbac-proxy-metric/0.log" Jul 07 18:36:52.016081 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:52.016056 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b776ec07-9a59-4ed8-9f3a-7880c377d46a/prom-label-proxy/0.log" Jul 07 18:36:52.057637 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:52.057610 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b776ec07-9a59-4ed8-9f3a-7880c377d46a/init-config-reloader/0.log" Jul 07 18:36:52.226285 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:52.226166 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-tkg2m_ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1/kube-state-metrics/0.log" Jul 07 18:36:52.274581 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:52.274555 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-tkg2m_ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1/kube-rbac-proxy-main/0.log" Jul 07 18:36:52.316629 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:52.316599 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-tkg2m_ed8ec45f-7e9f-4b47-8f7d-09813c73d4d1/kube-rbac-proxy-self/0.log" Jul 07 18:36:52.367695 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:52.367660 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-74cbbd4d86-rjp57_9e8864ce-4a84-4e3c-b838-a1a489470e5a/metrics-server/0.log" Jul 07 18:36:52.432533 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:52.432500 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-9f7gk_40a8daa5-ff78-4901-8230-f4d32a6ffd27/monitoring-plugin/0.log" Jul 07 18:36:52.480118 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:52.480042 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gws7k_6dbc6247-19f3-4e77-b809-18b641f4bdf9/node-exporter/0.log" Jul 07 18:36:52.519639 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:52.519600 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gws7k_6dbc6247-19f3-4e77-b809-18b641f4bdf9/kube-rbac-proxy/0.log" Jul 07 18:36:52.553100 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:52.553068 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gws7k_6dbc6247-19f3-4e77-b809-18b641f4bdf9/init-textfile/0.log" Jul 07 18:36:53.025408 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.025371 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2bb751ed-3254-44f6-ac51-d2f6246c6cd4/prometheus/0.log" Jul 07 18:36:53.077766 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.077723 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2bb751ed-3254-44f6-ac51-d2f6246c6cd4/config-reloader/0.log" Jul 07 18:36:53.126461 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.126427 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2bb751ed-3254-44f6-ac51-d2f6246c6cd4/thanos-sidecar/0.log" Jul 07 18:36:53.181535 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.181510 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2bb751ed-3254-44f6-ac51-d2f6246c6cd4/kube-rbac-proxy-web/0.log" Jul 07 18:36:53.234340 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.234312 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2bb751ed-3254-44f6-ac51-d2f6246c6cd4/kube-rbac-proxy/0.log" Jul 07 18:36:53.292654 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.292617 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2bb751ed-3254-44f6-ac51-d2f6246c6cd4/kube-rbac-proxy-thanos/0.log" Jul 07 18:36:53.373173 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.373140 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2bb751ed-3254-44f6-ac51-d2f6246c6cd4/init-config-reloader/0.log" Jul 07 18:36:53.539402 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.539360 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t248s/perf-node-gather-daemonset-9kpdg"] Jul 07 18:36:53.539839 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.539820 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="678edfb4-f4ee-41e4-af93-f2a741db1657" containerName="istio-proxy" Jul 07 18:36:53.539964 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.539842 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="678edfb4-f4ee-41e4-af93-f2a741db1657" containerName="istio-proxy" Jul 07 18:36:53.539964 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.539855 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20ddc125-fd2e-408a-9750-e817f2d9bf6c" containerName="istio-proxy" Jul 07 18:36:53.539964 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.539865 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="20ddc125-fd2e-408a-9750-e817f2d9bf6c" containerName="istio-proxy" Jul 07 18:36:53.539964 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.539879 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20ddc125-fd2e-408a-9750-e817f2d9bf6c" containerName="istio-proxy" Jul 07 18:36:53.539964 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.539888 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="20ddc125-fd2e-408a-9750-e817f2d9bf6c" containerName="istio-proxy" Jul 07 18:36:53.539964 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.539934 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd00ccaa-2e99-442d-83ff-9e776a01255c" containerName="maas-api" Jul 07 18:36:53.539964 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.539943 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd00ccaa-2e99-442d-83ff-9e776a01255c" containerName="maas-api" Jul 07 18:36:53.539964 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.539957 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="678edfb4-f4ee-41e4-af93-f2a741db1657" containerName="istio-proxy" Jul 07 18:36:53.539964 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.539965 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="678edfb4-f4ee-41e4-af93-f2a741db1657" containerName="istio-proxy" Jul 07 18:36:53.540401 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.539990 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d12baac8-ff09-4c0c-879a-0914c5dcaef3" containerName="maas-api" Jul 07 18:36:53.540401 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.539998 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d12baac8-ff09-4c0c-879a-0914c5dcaef3" containerName="maas-api" Jul 07 18:36:53.540401 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.540095 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="20ddc125-fd2e-408a-9750-e817f2d9bf6c" containerName="istio-proxy" Jul 07 18:36:53.540401 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.540107 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="20ddc125-fd2e-408a-9750-e817f2d9bf6c" containerName="istio-proxy" Jul 07 18:36:53.540401 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.540117 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="678edfb4-f4ee-41e4-af93-f2a741db1657" containerName="istio-proxy" Jul 07 18:36:53.540401 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.540125 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd00ccaa-2e99-442d-83ff-9e776a01255c" containerName="maas-api" Jul 07 18:36:53.540401 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.540136 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="678edfb4-f4ee-41e4-af93-f2a741db1657" containerName="istio-proxy" Jul 07 18:36:53.540401 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.540147 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d12baac8-ff09-4c0c-879a-0914c5dcaef3" containerName="maas-api" Jul 07 18:36:53.543584 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.543518 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t248s/perf-node-gather-daemonset-9kpdg" Jul 07 18:36:53.551868 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.551847 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t248s\"/\"kube-root-ca.crt\"" Jul 07 18:36:53.552034 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.552014 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t248s\"/\"default-dockercfg-ml2d6\"" Jul 07 18:36:53.552470 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.552455 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t248s\"/\"openshift-service-ca.crt\"" Jul 07 18:36:53.580489 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.580456 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t248s/perf-node-gather-daemonset-9kpdg"] Jul 07 18:36:53.632730 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.632703 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-rxn9v_4cc296ee-0400-4cc0-b79a-e21d0cc0929c/prometheus-operator-admission-webhook/0.log" Jul 07 18:36:53.720430 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.720383 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ab3216fa-c072-47d1-80e5-466d9d73e5a8-proc\") pod \"perf-node-gather-daemonset-9kpdg\" (UID: \"ab3216fa-c072-47d1-80e5-466d9d73e5a8\") " pod="openshift-must-gather-t248s/perf-node-gather-daemonset-9kpdg" Jul 07 18:36:53.720621 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.720465 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ab3216fa-c072-47d1-80e5-466d9d73e5a8-podres\") pod \"perf-node-gather-daemonset-9kpdg\" (UID: \"ab3216fa-c072-47d1-80e5-466d9d73e5a8\") " pod="openshift-must-gather-t248s/perf-node-gather-daemonset-9kpdg" Jul 07 18:36:53.720621 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.720496 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckbx6\" (UniqueName: \"kubernetes.io/projected/ab3216fa-c072-47d1-80e5-466d9d73e5a8-kube-api-access-ckbx6\") pod \"perf-node-gather-daemonset-9kpdg\" (UID: \"ab3216fa-c072-47d1-80e5-466d9d73e5a8\") " pod="openshift-must-gather-t248s/perf-node-gather-daemonset-9kpdg" Jul 07 18:36:53.720621 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.720593 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab3216fa-c072-47d1-80e5-466d9d73e5a8-sys\") pod \"perf-node-gather-daemonset-9kpdg\" (UID: \"ab3216fa-c072-47d1-80e5-466d9d73e5a8\") " pod="openshift-must-gather-t248s/perf-node-gather-daemonset-9kpdg" Jul 07 18:36:53.720733 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.720625 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab3216fa-c072-47d1-80e5-466d9d73e5a8-lib-modules\") pod \"perf-node-gather-daemonset-9kpdg\" (UID: \"ab3216fa-c072-47d1-80e5-466d9d73e5a8\") " pod="openshift-must-gather-t248s/perf-node-gather-daemonset-9kpdg" Jul 07 18:36:53.821305 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.821187 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ab3216fa-c072-47d1-80e5-466d9d73e5a8-podres\") pod \"perf-node-gather-daemonset-9kpdg\" (UID: \"ab3216fa-c072-47d1-80e5-466d9d73e5a8\") " pod="openshift-must-gather-t248s/perf-node-gather-daemonset-9kpdg" Jul 07 18:36:53.821305 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.821256 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckbx6\" (UniqueName: \"kubernetes.io/projected/ab3216fa-c072-47d1-80e5-466d9d73e5a8-kube-api-access-ckbx6\") pod \"perf-node-gather-daemonset-9kpdg\" (UID: \"ab3216fa-c072-47d1-80e5-466d9d73e5a8\") " pod="openshift-must-gather-t248s/perf-node-gather-daemonset-9kpdg" Jul 07 18:36:53.821552 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.821324 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab3216fa-c072-47d1-80e5-466d9d73e5a8-sys\") pod \"perf-node-gather-daemonset-9kpdg\" (UID: \"ab3216fa-c072-47d1-80e5-466d9d73e5a8\") " pod="openshift-must-gather-t248s/perf-node-gather-daemonset-9kpdg" Jul 07 18:36:53.821552 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.821342 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab3216fa-c072-47d1-80e5-466d9d73e5a8-lib-modules\") pod \"perf-node-gather-daemonset-9kpdg\" (UID: \"ab3216fa-c072-47d1-80e5-466d9d73e5a8\") " pod="openshift-must-gather-t248s/perf-node-gather-daemonset-9kpdg" Jul 07 18:36:53.821552 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.821373 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ab3216fa-c072-47d1-80e5-466d9d73e5a8-podres\") pod \"perf-node-gather-daemonset-9kpdg\" (UID: \"ab3216fa-c072-47d1-80e5-466d9d73e5a8\") " pod="openshift-must-gather-t248s/perf-node-gather-daemonset-9kpdg" Jul 07 18:36:53.821552 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.821390 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ab3216fa-c072-47d1-80e5-466d9d73e5a8-proc\") pod \"perf-node-gather-daemonset-9kpdg\" (UID: \"ab3216fa-c072-47d1-80e5-466d9d73e5a8\") " pod="openshift-must-gather-t248s/perf-node-gather-daemonset-9kpdg" Jul 07 18:36:53.821552 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.821436 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ab3216fa-c072-47d1-80e5-466d9d73e5a8-proc\") pod \"perf-node-gather-daemonset-9kpdg\" (UID: \"ab3216fa-c072-47d1-80e5-466d9d73e5a8\") " pod="openshift-must-gather-t248s/perf-node-gather-daemonset-9kpdg" Jul 07 18:36:53.821552 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.821463 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab3216fa-c072-47d1-80e5-466d9d73e5a8-sys\") pod \"perf-node-gather-daemonset-9kpdg\" (UID: \"ab3216fa-c072-47d1-80e5-466d9d73e5a8\") " pod="openshift-must-gather-t248s/perf-node-gather-daemonset-9kpdg" Jul 07 18:36:53.821552 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.821524 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab3216fa-c072-47d1-80e5-466d9d73e5a8-lib-modules\") pod \"perf-node-gather-daemonset-9kpdg\" (UID: \"ab3216fa-c072-47d1-80e5-466d9d73e5a8\") " pod="openshift-must-gather-t248s/perf-node-gather-daemonset-9kpdg" Jul 07 18:36:53.832063 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.832029 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckbx6\" (UniqueName: \"kubernetes.io/projected/ab3216fa-c072-47d1-80e5-466d9d73e5a8-kube-api-access-ckbx6\") pod \"perf-node-gather-daemonset-9kpdg\" (UID: \"ab3216fa-c072-47d1-80e5-466d9d73e5a8\") " pod="openshift-must-gather-t248s/perf-node-gather-daemonset-9kpdg" Jul 07 18:36:53.854028 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.853986 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t248s/perf-node-gather-daemonset-9kpdg" Jul 07 18:36:53.926515 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.926389 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-568b77c849-rc4mb_4a3bebfc-1f1e-4bb2-a02a-7174390c38b5/thanos-query/0.log" Jul 07 18:36:53.974514 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:53.974478 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-568b77c849-rc4mb_4a3bebfc-1f1e-4bb2-a02a-7174390c38b5/kube-rbac-proxy-web/0.log" Jul 07 18:36:54.006389 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:54.006302 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t248s/perf-node-gather-daemonset-9kpdg"] Jul 07 18:36:54.009277 ip-10-0-136-192 kubenswrapper[2583]: W0707 18:36:54.009248 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podab3216fa_c072_47d1_80e5_466d9d73e5a8.slice/crio-b7aa3c6c82fe79467a99980827ccc75c3fbd006c6524847a366361e38c33feb7 WatchSource:0}: Error finding container b7aa3c6c82fe79467a99980827ccc75c3fbd006c6524847a366361e38c33feb7: Status 404 returned error can't find the container with id b7aa3c6c82fe79467a99980827ccc75c3fbd006c6524847a366361e38c33feb7 Jul 07 18:36:54.019791 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:54.019765 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-568b77c849-rc4mb_4a3bebfc-1f1e-4bb2-a02a-7174390c38b5/kube-rbac-proxy/0.log" Jul 07 18:36:54.071615 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:54.071540 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-568b77c849-rc4mb_4a3bebfc-1f1e-4bb2-a02a-7174390c38b5/prom-label-proxy/0.log" Jul 07 18:36:54.114261 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:54.114224 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t248s/perf-node-gather-daemonset-9kpdg" event={"ID":"ab3216fa-c072-47d1-80e5-466d9d73e5a8","Type":"ContainerStarted","Data":"59d2ed7225404352da9f9505c51b45e5e8dd2f00cde7a87c6593f14837030434"} Jul 07 18:36:54.114452 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:54.114279 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t248s/perf-node-gather-daemonset-9kpdg" event={"ID":"ab3216fa-c072-47d1-80e5-466d9d73e5a8","Type":"ContainerStarted","Data":"b7aa3c6c82fe79467a99980827ccc75c3fbd006c6524847a366361e38c33feb7"} Jul 07 18:36:54.114452 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:54.114355 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-t248s/perf-node-gather-daemonset-9kpdg" Jul 07 18:36:54.119415 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:54.119389 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-568b77c849-rc4mb_4a3bebfc-1f1e-4bb2-a02a-7174390c38b5/kube-rbac-proxy-rules/0.log" Jul 07 18:36:54.169032 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:54.168997 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-568b77c849-rc4mb_4a3bebfc-1f1e-4bb2-a02a-7174390c38b5/kube-rbac-proxy-metrics/0.log" Jul 07 18:36:55.893450 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:55.893417 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wqfm4_997df85c-7632-47b5-a8a6-1c37f3346251/console-operator/2.log" Jul 07 18:36:55.898612 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:55.898582 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wqfm4_997df85c-7632-47b5-a8a6-1c37f3346251/console-operator/3.log" Jul 07 18:36:57.334058 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:57.334025 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-ppncs_2a31b62e-e3ad-4c70-a495-37a30122dc22/volume-data-source-validator/0.log" Jul 07 18:36:58.579797 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:58.579766 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-847hz_4e09f673-9441-4afe-a258-85b6b5203de8/dns/0.log" Jul 07 18:36:58.627114 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:58.627082 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-847hz_4e09f673-9441-4afe-a258-85b6b5203de8/kube-rbac-proxy/0.log" Jul 07 18:36:58.810015 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:58.809979 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p4mnt_e5c3b9f0-ea6c-4262-b4e7-88945edc9f6f/dns-node-resolver/0.log" Jul 07 18:36:59.461589 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:59.461543 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-77c4ff87fd-tjwtz_4cb6eb72-4869-400a-a56d-d712f4c8ff83/registry/0.log" Jul 07 18:36:59.499487 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:36:59.499450 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dm6hm_d5ea083a-f7fb-4e82-a02a-b930204910b6/node-ca/0.log" Jul 07 18:37:00.130209 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:37:00.130175 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-t248s/perf-node-gather-daemonset-9kpdg" Jul 07 18:37:00.153356 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:37:00.153296 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-t248s/perf-node-gather-daemonset-9kpdg" podStartSLOduration=7.153274045 podStartE2EDuration="7.153274045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 18:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 18:36:54.147795805 +0000 UTC m=+2563.230399508" watchObservedRunningTime="2026-07-07 18:37:00.153274045 +0000 UTC m=+2569.235877750" Jul 07 18:37:00.746938 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:37:00.746871 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4r59rt_6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2/istio-proxy/0.log" Jul 07 18:37:00.905521 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:37:00.905488 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-j4f4p_27c52354-e3bb-47b5-94ca-6ec736c41d3f/discovery/0.log" Jul 07 18:37:01.213671 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:37:01.213641 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d876b7659-t4r6h_4d9f9071-7f2b-4944-88fc-130da6a92ba5/router/0.log" Jul 07 18:37:02.266181 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:37:02.266147 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dr6hd_686b9d0e-9891-4e46-b78b-c5030ceffa06/serve-healthcheck-canary/0.log" Jul 07 18:37:03.043646 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:37:03.043615 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-89b5k_44101f7b-1922-44b1-9042-8a5bdc474c87/kube-rbac-proxy/0.log" Jul 07 18:37:03.102048 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:37:03.102016 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-89b5k_44101f7b-1922-44b1-9042-8a5bdc474c87/exporter/0.log" Jul 07 18:37:03.196762 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:37:03.196730 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-89b5k_44101f7b-1922-44b1-9042-8a5bdc474c87/extractor/0.log" Jul 07 18:37:04.486955 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:37:04.486922 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4r59rt_6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2/istio-proxy/0.log" Jul 07 18:37:04.596865 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:37:04.596835 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-j4f4p_27c52354-e3bb-47b5-94ca-6ec736c41d3f/discovery/0.log" Jul 07 18:37:04.822098 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:37:04.822062 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d876b7659-t4r6h_4d9f9071-7f2b-4944-88fc-130da6a92ba5/router/0.log" Jul 07 18:37:06.214016 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:37:06.213984 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4r59rt_6f1a1c19-af97-46c4-baf9-0aef3ae8d7f2/istio-proxy/0.log" Jul 07 18:37:06.321849 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:37:06.321806 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-j4f4p_27c52354-e3bb-47b5-94ca-6ec736c41d3f/discovery/0.log" Jul 07 18:37:06.567166 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:37:06.567137 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d876b7659-t4r6h_4d9f9071-7f2b-4944-88fc-130da6a92ba5/router/0.log" Jul 07 18:37:06.916453 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:37:06.916335 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76cd65fb98-6cfbr_1bb166ca-6b5b-4ed8-af7c-97d60eed5e94/manager/0.log" Jul 07 18:37:06.944328 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:37:06.944301 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76cd65fb98-6cfbr_1bb166ca-6b5b-4ed8-af7c-97d60eed5e94/copy-manifests/0.log" Jul 07 18:37:08.679191 ip-10-0-136-192 kubenswrapper[2583]: I0707 18:37:08.679166 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-58f6946598-w6cps_ff4c870c-40bd-43b1-91c3-1f437a5349c3/manager/0.log"